Player is loading...
Supporting the Scala Ecosystem: Stories from the Line
Justin Pihony, Lightbend
Thursday, June 13, 2019 · 12:16 p.m. · 36m 40s
At Lightbend we have handled over 10,000 customer support cases ranging from simple Scala compilation issues to complex Akka performance puzzles. Supporting Scala and its largest libraries has resulted in some really interesting issues, such as: a seemingly valid cast leading to a ClassCastException; handling external communication with Akka Cluster; a subtle workaround to a SecurityException bug in Play. Required Knowledge: No previous knowledge is required as this should be for all experience levels. Justin is a software journeyman, continuously learning and honing his skills. He is currently using his knowledge to provide developer support at Lightbend. As much as he loves to learn, he also loves to spread his knowledge through teaching and helping others. He has authored three online courses for Pluralsight, including a Spark Fundamentals one, is one of the top Spark answerers on StackOverflow, and organizes the Pittsburgh Scala meetups.
Embed
Transcriptions
Note: this content has been automatically generated.
00:00:00
uh so this is a supporting this gal ecosystem stories from line um
00:00:06
is basically going to be a i'm actually out about quick about myself and other
00:00:12
explain kind of what to talk about so quick about myself is i'm a uh
00:00:17
problem solving enthusiastically band which really means i'm
00:00:20
the developer support managers um so um i always
00:00:25
uh make it a can to tell people what they can to stack overflow plus plus uh
00:00:28
the harder questions you can stack overflow that's the kind of stuff that we get so it's
00:00:33
it's a a fun job in my opinion we get to solve a bunch of different problems
00:00:39
and that we're gonna be uh uh seeing is uh some of
00:00:42
the more interesting cases uh realistically it's going to range from um
00:00:48
kind of a more complex cases to some simple ones just so
00:00:52
that it covers the crowd pretty well to um uh i'm also a
00:00:58
spatter of knowledge so i'm up for us like arthur stack overflow answer and my my answer the questions
00:01:03
true like bent so um so uh i really love to learn where i love to teach as well
00:01:09
as i love to learn so both sides of it uh uh which is essentially what i'm here today
00:01:16
so the scenarios we're going to uh be covering are going to be a a skylight pass
00:01:22
class cast hustler so that of a puzzle or whatever love puzzles um
00:01:30
taco cluster external communication us something we've gotten a fair amount of so um in
00:01:36
fact i think internally we have a a frequently asked question if the article for
00:01:42
as well as a a subtle work around for a place security exception
00:01:46
uh no such method which will but become kind of something else will see it never get to it
00:01:52
a scallop value load order um and sorting out
00:01:57
those last two realistically i added them in there just
00:02:01
to allow for time they're gonna be the easier items so if we don't get to them
00:02:05
wanted in the world the first for our um the most interesting in my opinion
00:02:10
um i yeah so moving on so the first item is the class cast puzzle or so we had a case come in
00:02:20
where they uh had a a simple method just like this so
00:02:25
they wanna say they want to create their own add method it's generic um
00:02:29
and the person you're gonna see that probably makes you kinda question is this uh
00:02:35
has instance of um but you can see that there there it was it is uh
00:02:40
for numeric so realistically um whatever casting it you should work so
00:02:47
sure you're gonna say okay of course related class cast exception you're doing as instance of but again
00:02:53
it's forcing numeric so uh that realistically should work let's find out what happens
00:02:58
so whenever we pass in an integer
00:03:02
it had so so if you want thing it is a is a and you can see it's adding one basically so the method
00:03:08
would be better name just add one but either way whenever we
00:03:12
had one to one we get to works perfectly we're happy now
00:03:18
what happens whenever we add a a wondered zero so now
00:03:23
we get a class cast exception saying well like this with a class that
00:03:29
exception where uh the integer cannot be cast to java doubling that double or so
00:03:39
it's kind of like okay why is that happening what what kind of see um
00:03:45
one to as instance of t. v. i mean it's a double i mean
00:03:49
it's numeric it should work right um and even if you go and do that
00:03:55
one as instance of double it works you in the rubble it it works so why whenever we're passing
00:04:02
an in the the generic way is uh is it not working i so let's take a little deeper um
00:04:11
here you can look at kind of what this boils down to
00:04:16
um which is going to be this um
00:04:22
non make numeric but the key important thing here is boxes run time boxed integer so
00:04:29
b. compiler behind the scenes is kind of know lapping this one it into an
00:04:33
integer so it's forcing in saying this one is an integer and ultimately then whenever you
00:04:41
go to this plus method you're passing you the uh
00:04:45
um generic is saying hey this is gonna be a double so you're getting it say
00:04:49
a double with the double yet which is box it to nature it's gonna now crash
00:05:00
okay so now that we have a little bit more idea of what's going on we can even simplify this method we can just say
00:05:09
look let's get rid of just the at this in part of it we can even just get go down to
00:05:13
creating a generic one ad instance of t. and the elements we're just gonna
00:05:18
throw away we more care about the actual type that's been passed in so still
00:05:24
we end up with this box to integer and ultimately whenever you try to cast it
00:05:31
to a double then you end up with this phone box the double your uh your
00:05:38
yeah so you're you're box the double three d. and um whenever you on box
00:05:44
i think i may have title but i think that's lesbian box to integer but uh
00:05:49
or boxed integer but either way uh it you you end up with the same crash so ultimately
00:05:56
the problem here is just like parades or so behind the
00:05:59
scenes the reason we're able just uh uh reproduce the same problem
00:06:04
uh with that simple method it is because behind the scenes it's really a one
00:06:08
as instance of object so even though you have the types and everything in there
00:06:13
and it looks like it would work the compiler things it should work um it's because it behind the
00:06:18
scenes it's come by i think it should work because it's saying hey it's it's it's a object so
00:06:24
i'm not gonna get into this part um uh output the slides up under
00:06:28
for anybody who's really curious i wanna put this out there as a uh uh
00:06:32
fun thing to add to it uh this case was i i was helped with uh
00:06:38
by jason's are so one of the benefits of working like vendors you work with some really
00:06:42
really intelligent people so he kind a talking some stuff about hey you the the java p.
00:06:48
you can basically uh i inspect budget be
00:06:52
java using java p. you your class and everything
00:06:55
and he can blow it down to just doing this these um
00:06:59
as instance of on generics and it'll you'll be able to see that
00:07:03
where this blows up is finally on a check cast so um for anybody who goes ahead and brands
00:07:10
that that like as opposed to in you'll be able to run those and the rebel and he'll get up
00:07:15
crazy amount of stuff but um you'll see that there's eventually check asked and that's where it was up
00:07:22
so next thing uh the pocket cluster external communications so uh
00:07:29
we've got our ah cluster it's running happily and we wanna
00:07:33
send the message from n. x. journal point an extra interactive
00:07:37
system or something and we wanna send it into our our cluster
00:07:43
one way you can i guess the the question is how can you do this and um
00:07:48
one way that had been promoted for awhile is to use a cluster klein
00:07:54
so that's gonna be say hey i'm gonna wrap this actor and say it's cost a client
00:07:58
and behind here inside of your there's a a
00:08:01
cluster client receptionist and this will work um however
00:08:07
what's gonna happen is you're gonna be really tightly coupled
00:08:11
to your remote loading and the the the mixer the c.
00:08:15
realisation and talk a versions and everything between your clients
00:08:19
and your um your your source there that they're gonna be
00:08:23
require to be pretty much the same thing so realistically not a good choice
00:08:28
so that's that's work lines came in and said hey we wanna use something like
00:08:31
streamer f.'s or something like that however again whenever you get a remote in interior
00:08:36
work but you're gonna be tightly coupled as well as you run the risk of
00:08:41
uh this uh kind of connection issues or something like that because
00:08:46
they're not in the same they're not actually forced to be in the
00:08:49
same system that's where a arcos clustering is gonna be a little
00:08:54
bit better so so what's what's better than that so nowadays we've got
00:08:59
um ah k. h. t. t. p. a. and pocket g. r. p. c. uh i'm not get
00:09:04
too much into the the the entrails of those
00:09:08
'cause that's comedian talk of in its own um however
00:09:11
it the key here is that you're creating a uh an interface that this message is gonna
00:09:16
go to it can be now now you're you're not a couple of your client being um
00:09:23
park or anything like that it can be anything that can talk to uh
00:09:26
this ten interface so it through like h. t. t. p. or anything like that
00:09:31
as well as um both pocket your p. c. and ah k. h. t. t. p.
00:09:36
both allow for easy strain so whenever customer said hey we want
00:09:40
you stream has agreed to be an okay g. r. p. c. still
00:09:44
allow you that capability so uh those are the ways to go um i don't know if it's on the type
00:09:50
or actually it has moved into the documentation it's not necessarily in big bold letters but um we do say hey
00:09:56
you should think about instead of using cluster by you should be using maybe um h. t. v. and your p. c.
00:10:04
okay the next item is play security exception so this one actually
00:10:10
is a support case that came in but realistically is referencing a
00:10:15
play framework gives you that is actually still outstanding but um i believe
00:10:19
the but we it should might be fixed uh in the upcoming play version just because of our
00:10:26
upgrades in our case you speak about what it is is it's gonna say okay
00:10:30
be a a java you till collects in parallel stream raises a security exception
00:10:36
when call ins inside the controllers action so let's see what this looks like
00:10:41
i'm pretty simple method the the you can reproduce this really simply to say
00:10:46
okay the uh i cut out like the craft around it to this is
00:10:50
out of control or you've got a simple get to get barges return strings
00:10:54
uh and behind the scenes the the
00:10:57
problem is this supply a sink method um
00:11:05
whenever you had that supply i think method even though i mean it's simple code it should work everybody's happy um
00:11:12
realistically the problem is that behind the scenes that
00:11:15
using a a four join common poll which is
00:11:20
really really locked down and whenever you go to use it you end up with that security exception so
00:11:27
again it's about it's about the inside of play but uh we need to fix it any customers are heading
00:11:33
this we've been we need to make them happy right so we need to work around off while the fixes
00:11:39
coming out i mean the version of play that is supposedly fixes it isn't even out yet and it's a
00:11:43
major version release customers are gonna stay on all versions for a while we know that so far with that's it
00:11:50
luckily a supply a think can have a context past
00:11:55
and so you can simply inside of your um controller method
00:12:00
you can simply inject b. d. for execution context and with this is gonna do
00:12:06
is this is gonna a pass in the uh uh ca the execution context that
00:12:12
play uses so it's simple all you do is you just injected and and you don't have to worry about it you
00:12:18
don't have to actually pass it in the injector will feel the plate frame working everything would deal with it for you um
00:12:25
and then the supply a think takes has a in
00:12:28
overload where you can actually pass that context and and it'll
00:12:32
override that common pole and now it's a lot more
00:12:35
lax the only thing to uh take into account here is
00:12:41
um there's a this this bit we take you to the play framework page about
00:12:46
creating a custom execution context so whenever i say it's
00:12:50
really simple tat pass in the uh execution context that
00:12:54
plays using keep in mind plays using that so if you're doing any kind of heavy blocking inside of here
00:13:00
that and the realistically you may wanna create a custom execution context but if it's a pretty simple method uh you can just
00:13:07
the simple add into the constructor use it you're done and you fix the problem
00:13:14
okay so the next thing this is a bit of a duplicate i did a round a couple years ago about um
00:13:21
supporting sparks uh but this uh is a story about
00:13:25
a no such method in spark that uh i think
00:13:29
it's still it up the the versions and everything it's
00:13:32
it's the a bit old but it still is very relevant
00:13:36
i we get tons of problems around this similar should so what is the issue
00:13:41
um actually before we get to the us you let's let's the uh dig into what
00:13:45
i know such method is um the key here is it's going to be a a
00:13:50
a class no longer has a definition of that method and at run time
00:13:54
the definition has a in compatibility each the then compatibility has changed it so now
00:14:03
we get to uh the actual issue here
00:14:06
being spark one for one uses math three or
00:14:11
a three one one waits as it turns out doesn't have the natural
00:14:16
method so in this case it's a customary would using different versions of spark
00:14:22
um and they end up with a dependency collision so um
00:14:26
it was compiled against a spark one for would uh x. ray
00:14:32
for guide we that was sparked one three or something
00:14:35
like that and then it was run against spark one for
00:14:40
so compilation and everything's happy but whatever goes into the run time
00:14:44
it using a different version of spark endicott and it crashes so um
00:14:51
how do we solve this problem uh the first will cover the spark specific ways here which
00:14:56
blend into one of your energy pansy problems in general but um first park you can upgrade spark
00:15:04
i mean it's can be simple enough however again if you're working with uh come a
00:15:08
larger company uh upgrading versions can be a bit of a pain so uh do that
00:15:16
it's only gonna be available if that's why added the
00:15:18
if you're lucky because i mean what people don't allow that
00:15:24
so the other thing is we can enforce library low order and spark we can say hey
00:15:29
we're gonna add to the class button we're gonna say this is
00:15:32
the the version of commons math that we're gonna be using um
00:15:37
it's a bit of a brute force but it can work the other option that i think works in general whenever you run into a a
00:15:43
a dependency collisions that for whatever reason maybe you have to have different
00:15:48
versions is shavings about s. p. t. and may even uh i have
00:15:52
the ability to shade and and for anybody who's not familiar with it uh we're saving is is it's
00:15:57
gonna say hey i have a different version and whenever i kind of built into my package it's going to
00:16:03
kind of um obfuscate the name so that the two
00:16:07
packages can coincide together like what i'm what i've read
00:16:10
in it it's allowed to uh to um to run
00:16:14
because the run time sees as a different library altogether
00:16:20
um however like i said this is gonna lead to a separate
00:16:24
it's you that is more overarching of what we get within uh oh
00:16:29
support also the time which is dependency chaos uh sometimes you'll get it as no such methods such as this
00:16:37
other times where you're gonna end up with is uh we've had schuyler
00:16:41
is use where okay we have a type mismatch on some generic yet
00:16:46
by all means it should be working in in this case it's always being caught at compilation but
00:16:51
they just can't figure out what's going on um turns out that it's a uh uh the uh
00:16:56
in compatibility between a different binary is i think all in that case all they need to use clean but
00:17:02
these can be really difficult things that you're like scratching your head i i i don't understand what's going on here so again
00:17:11
keeping your dependencies happy is the way to go
00:17:13
um pockets of a cluster has a way to
00:17:18
detect possible incompatible versions and the class paths so it will actually
00:17:23
print this met message and you're a lot i think it's a warning it's not gonna be in error
00:17:28
anything but a warning that saying hey you've got different versions of ah cut grass or cluster so ultimately
00:17:35
there is a kind of problems that you can run into that
00:17:37
are all based around um dependencies so making sure that you're managing your
00:17:43
dependencies as best as you can sometimes like in the case of spark
00:17:46
it's hidden from you when you have to dig into it um but
00:17:50
uh that as best as you can making sure that your dependencies are all in sync
00:17:54
with the the libraries are using and everything is is critical to avoiding head scratching books
00:18:01
okay um so be it next item is a a value load orders
00:18:08
so inside of scale well we have a the the problem statement of
00:18:15
when starting a program where and key is your scale objects are not initialise yet
00:18:20
it fails with a null pointer to exception because they're trying to access the out
00:18:26
um so this is probably something people are where um
00:18:32
but uh in this case you can market as lazy so they know
00:18:37
hey up potential solution is change too lazy about wonder windup solving your problem
00:18:42
um however in their case they need to use sometimes use of our toolbar because it may change um
00:18:48
so just changing it to lazy isn't necessarily a problem so
00:18:53
again we know lazy you could solve the problem but in their case maybe not so
00:18:59
the next thing to do is um
00:19:01
while objects are lazy they're instantiated leisurely
00:19:06
um you can instantiate them um in
00:19:12
uh uh it at at the beginning of your program so the first
00:19:15
reference i mean this it will care as everybody i'm sure is aware
00:19:19
a style objects are singleton so the first time you reference it's gonna
00:19:24
load it and it's gonna cash it in and it's the you're gonna
00:19:26
be find from then on out so if you run into a problem
00:19:30
we're loading order is your issue then maybe you can just call it
00:19:35
um at the beginning of your of your program and to say i'm referencing it and just to take it off
00:19:41
so those are all pretty simple things um however
00:19:45
the reason i brought this up is because toll uh at the
00:19:49
end um uh jason's out on this one too he pointed out
00:19:53
a a a a a toll within scholar that i found to
00:19:56
be really really useful so were or were rescued here and that is
00:20:05
well uh i had slide just just exactly the problem first then we'll see where the tolling comes into play um
00:20:13
so a pretty simple way to exhibit it is saying hey we're gonna have this uh
00:20:19
o. one and o. two and they be o. two kinds of referencing
00:20:26
no one's objects so whenever we come down yeah we tried the prep
00:20:31
oh one die a um then it will end up printing or not
00:20:40
so we're gonna see where this is actually happening by throwing in this terrible prints tracts point that
00:20:46
print stack trace are gonna see that that uh initialisation order is kind of the problem where you see
00:20:52
oh to see is call no one and basically uh one
00:20:56
is in full initialised yet so the whole thing was up
00:21:02
uh but like i said tone to the rescue so again we we know the ways to solve this but
00:21:07
the important reason that i bring it up is if you add this
00:21:10
uh the compiler chuck x. check in it then you're going to get
00:21:17
a a a a a error here scallop on initialise
00:21:21
fewer and it's gonna tell you exactly where your problem is
00:21:25
so the reason i point that out is you don't necessarily want to run those brian in production
00:21:31
with that uh check in place however in your
00:21:34
c. i. or just a regular uh development by me
00:21:38
if you just add that simply then you might save yourself headache that a a headache down the road just because all of
00:21:44
a sudden hey uh the the style compiler is now gonna
00:21:48
catch these problems for me it's gonna tell me where they are
00:21:51
and i can fix them before it becomes our our problem in run time
00:21:58
so again nice little happy thing there to use the next thing is uh sorting a map
00:22:07
so problem statement here is when we include more than three sorting
00:22:12
columns and i request the sorting order of the columns are not persisted
00:22:16
so in this case uh the customer centres code that
00:22:20
looks like this you don't necessarily need a parsed for
00:22:24
folding or anything like that uh uh it this is just to show you what the problem is and in this case
00:22:31
but the problem is here so this uh accumulator is a map of the string to string
00:22:38
um and so in their mind that they're adding
00:22:42
elements in order so they're saying okay this accumulator added this and it's in the the correct sorted order
00:22:48
that i want and then i read this and this and this and it it should work right um bye
00:22:55
as many of you may or may not know um maps are actually not sort of all
00:23:01
so we can actually like is that you don't need to care about the code that was there just the the basic problem because now we
00:23:07
can actually see it in a a more uh and the wrapper we
00:23:12
can just start adding things it's extremely reproduce double you just say okay
00:23:17
i mean it's into an empty map for the in integer to
00:23:19
string where it's just the integer to the string representation of itself
00:23:24
um and you say okay one to one two alright we're still in her purse on
00:23:29
order as we get the three were stolen or as we get to four and then it's
00:23:35
uh_huh and into for
00:23:45
okay uh that at the bottom all the sudden we had the
00:23:48
size and now five is inserted at the beginning so again max
00:23:54
the the they are built to be efficient and be able to just pass and
00:23:57
look things up so if you're expecting to treat this like a sequence and actually
00:24:03
uh see in sorted order anything like that you're out of luck
00:24:08
so in this case i guess that it's actually pretty simple um
00:24:11
it's more about knowing the a. p. i. um it's the same thing with the ah cluster that
00:24:16
we saw earlier uh just knowing the problems is
00:24:19
more than half half of the uh uh the salute
00:24:23
they're saying hey i know this problem i know the a. p. i. i know what's out there uh and i can actually uh make use of it so in this
00:24:30
case there's actually something in the libraries gala collection immutable list map and now we do
00:24:35
the same thing that one two three four still in order we had the fifth one
00:24:41
and it's still gonna be in order um we can keep going through and everything's gonna be in it's still gonna work um
00:24:49
now the only thing to know about this class is
00:24:53
that becomes overhead so i mean at the scale out and
00:24:57
you've got the sorting everything that goes with it so
00:25:00
um there's a certain amount of um headaches that you might
00:25:04
uh have have to deal with if you're trying to run the set scale so in that case actually um
00:25:11
yeah we talked was not uh i wanna point out that uh
00:25:14
later today there is a a talk and immutable sequent sequential maps um
00:25:20
keeping order while house and i believe it's uh it's supposed to be a more
00:25:25
performance solutions so uh like to guess maybe checking that out if that's of interest you
00:25:33
uh that's um those are the different uh scenarios that i have here
00:25:39
um actually had expected to run a little bit longer than those um but
00:25:46
it's fine we can have a questions if not you have kind of a backup
00:25:48
story is no slightly anecdotal um uh but i do want to point out that
00:25:54
uh other support team as well as i think we have some engineering positions there but if if you
00:25:58
like kind of these problems uh we are hiring a
00:26:02
feel free to talk to me um for a whoops
00:26:11
uh you can if you like to have this level of support
00:26:16
and does your jason's dogs in your back pocket and uh myself
00:26:20
in all the play engineers and whoever else then come talk to us we have a bit of an uh will be happy to
00:26:27
uh but make your subscribers so you can get this level support plus points of other different parks
00:26:33
um the other thing is that you wanna buy now uh i'd like to learn from my talks a so feel free
00:26:41
to email me talked me directly or just use the apple and just let me know uh how you think at it
00:26:47
that said uh we you have some time so i can take questions or like i said i can
00:26:52
even go back and talk about kind of a a a separate case that's i found to be interesting
00:27:01
i think you eighteen questions i i i
00:27:14
okay uh so uh does anyone at so how many people here have a
00:27:20
if if no questions per se how many people here have heard of our pocket
00:27:27
right so uh we nodes in general that there's august rains and
00:27:30
i'll talk as we can add to add connections and and get
00:27:34
august rains in the streaming h. t. i. and everything like that from different libraries so
00:27:39
the most uh uh uh used library there are i believe it's still to date most used
00:27:45
is uh what was known as the kafka reactive strange but now it's all pocket kafka so
00:27:51
connecting your caucus streams to um uh our
00:27:56
constraints are actually it's not using kafka streams connecting
00:27:59
your kafka to a park uh so that you can have a streaming a. p. i. there
00:28:03
i mean to your are constraints um recently we added a
00:28:08
a a relatively recently we added support for a transaction all
00:28:13
uh and so obviously that we ran into here was
00:28:17
uh we had a customer who is really pounding it
00:28:21
uh and the they said okay well i have a a transaction
00:28:24
all source um facts let me see i can see shows uh
00:28:33
could show the code 'cause i don't believe there's anything
00:28:39
yeah i double checked before there's nothing a customer specific in it because it was just
00:28:45
a proof of concept so let me track this over to actually just change my earring
00:29:00
a a a yeah and
00:29:06
probably hard to see though to him she uses up um
00:29:19
sh sh sh rather than playing around with that might be a little hard
00:29:24
to see but um i'll go over here the the key item here is
00:29:28
uh this these little lines here where we say um this is one
00:29:36
transaction oh no this is the this is the consumer i'm sorry i directed the wrong
00:29:42
oh
00:29:50
there's a lot of crap
00:29:55
okay yeah here is the the code in and so you see that we have the
00:29:59
uh the transaction all source here so it's a pretty uh a simple a. p. i.
00:30:03
says hey i wanna have transaction all i say here's my source uh you're gonna do
00:30:08
some kind of propagation you're gonna add some stuff and you mapping and everything like that
00:30:13
and do it it's really simple it's just doing room really basic a and and just doing a simple map
00:30:19
them as it comes in and then i'm gonna output it to this transaction all sink and everything should be happy
00:30:26
however in the tests um and that gets to the other
00:30:35
scenario so in the tests uh they were being run and
00:30:39
uh it was being run through this uh this source
00:30:42
which is running at reed committed so all things that
00:30:46
are suggested for inside of the um cough a. p.
00:30:49
i. to say a makes or that transactions are happy
00:30:53
is we're gonna say h. we committed we're gonna run it through this uh this source we're gonna have transactions
00:30:59
and then we're gonna sit there and have this size and so basically the which we
00:31:04
have as the size of the list and the size of the set so that we
00:31:07
can say hey each one of these if that's the size of the set which is
00:31:11
distinct uh should be the same as the size of the list which is not distinct
00:31:17
so i have to go through it happy it's happy it's happy we get to about i don't
00:31:23
know uh sometimes four hundred thousand five hundred thousand messages and all the side and broom you start seeing
00:31:30
uh the size of the bill the set and let's start a d.
00:31:34
v. a. so with a little bit of digging we realised that um
00:31:39
what the actual problem was was that
00:31:44
the the consumer uh was a roll of
00:31:49
and did not receive uh the rebalancing so at some point uh it's just
00:31:56
some people were able to reproduce this on their machines and that was simply because
00:32:00
other machines were fast enough however on my machine i was i i was bombarding it with enough stuff i guess i don't know
00:32:07
but i was able to reproduce the senate with and what you
00:32:10
ultimately side was a as this was running through when printing thing is
00:32:14
uh all of a sudden it said hey i i slowed up and now all of a sudden i'm gonna run rebound
00:32:20
and so inside of the our pocket code once you dug into it you
00:32:24
found out okay i have this um this chunk and it would only um
00:32:33
yeah we don't need should could work um so inside of the inside of a pocket it was running it
00:32:39
was saying commit this transaction every so often i wasn't
00:32:43
necessarily giving you kind of granular control or whatever so
00:32:46
ultimately what would happen is they would out what the the the messages and it would end up
00:32:53
spitting them out and had it in her pocket had its own internal q. of these things
00:32:58
and say okay now we're gonna transact now we're gonna transact now we're gonna transact um however
00:33:04
every time there was a re balance it would go through and
00:33:08
you would end up seeing that um it wouldn't clear those jury
00:33:13
messages that were in its internal q. so now it had double
00:33:18
messages inside of this internal q. and whenever it did end up
00:33:23
uh finally get a real rebounds was done and everything it take back to the last point weights
00:33:30
was above these other messages and you ended up with double messed
00:33:36
um so i seems like okay could be a
00:33:40
simple solution however because um apart that is asynchronous
00:33:46
and everything um it became really tough problem because
00:33:49
the source and the sink don't necessarily know each other
00:33:54
so you can't necessarily say okay well this is rebound since i
00:33:58
got out of the source 'cause they're they're meant to be um
00:34:01
separated you don't want them to be necessarily couple together however in this case whenever you have a flow that is
00:34:08
a a sort of transaction of source followed by a transaction transaction
00:34:12
all saying can you wanna make sure that they're all together they
00:34:15
need to know about each other um so we actually dug into
00:34:18
we notice that kafka streams didn't have this problem and that's ultimately
00:34:23
because some of the stuff that they're doing is using global stay and and i believe there's a little bit of blocking at some
00:34:29
point um but we wanted to stay true to our nature of
00:34:32
asynchronous message message of and all that stuff that's reactive um so ultimately
00:34:38
uh_huh we ended up pushing out a case of the in
00:34:41
this case i it's more anecdotal about as an interesting story uh
00:34:45
because now if you go out and you grab the latest l. park uh it should just work but after all that they're getting
00:34:51
we found out okay we had to rebuild um internally be the source and sink so that
00:34:56
the kind of communicate with each other so that when that rebounds happens we're gonna clearer state
00:35:02
we're going to then whenever it it the the rebounds has repercussions things gonna say hey
00:35:08
we're gonna start again uh we have the same partition but all those things that we
00:35:13
thought were already posts were about to be post aren't there anymore we've cleared out everybody's happy
00:35:19
um i think they're robots of other little buttons you happen to find out the case but
00:35:23
again uh the it was definitely an interesting case and um
00:35:28
in my opinion uh i would love to run some benchmarks against
00:35:33
uh in this is totally me this is my opinion my or my own not necessarily like bands or anything um
00:35:38
by i would love to run some benchmarks against caucus trends in
00:35:42
our pocket streams because um like i said i noticed some their
00:35:46
their way of dealing with a padlock yes i'd with some blocking
00:35:49
and some global stay and things that are just going to slow down
00:35:52
so uh fully asynchronous um stream transaction i feel
00:35:58
like it it actually should speed along or even
00:36:01
faster then after strange itself um however you had pinpoints that uh like i said i just discussed
00:36:09
so uh that that's all i have and um if anybody has
00:36:14
any questions now go ahead otherwise uh thank you and again uh
Conference Program
A Tour of Scala 3
Martin Odersky, Professor EPFL, Co-founder Lightbend
June 11, 2019 · 5:15 p.m.
8333 views
A story of unification: from Apache Spark to MLflow
Reynold Xin, Databricks
June 12, 2019 · 9:15 a.m.
1267 views
In Types We Trust
Bill Venners, Artima, Inc
June 12, 2019 · 10:15 a.m.
1569 views
Creating Native iOS and Android Apps in Scala without tears
Zahari Dichev, Bullet.io
June 12, 2019 · 10:16 a.m.
2231 views
Techniques for Teaching Scala
Noel Welsh, Inner Product and Underscore
June 12, 2019 · 10:17 a.m.
1293 views
Future-proofing Scala: the TASTY intermediate representation
Guillaume Martres, student at EPFL
June 12, 2019 · 10:18 a.m.
1156 views
Metals: rich code editing for Scala in VS Code, Vim, Emacs and beyond
Ólafur Páll Geirsson, Scala Center
June 12, 2019 · 11:15 a.m.
4695 views
Akka Streams to the Extreme
Heiko Seeberger, independent consultant
June 12, 2019 · 11:16 a.m.
1552 views
Scala First: Lessons from 3 student generations
Bjorn Regnell, Lund Univ., Sweden.
June 12, 2019 · 11:17 a.m.
577 views
Cellular Automata: How to become an artist with a few lines
Maciej Gorywoda, Wire, Berlin
June 12, 2019 · 11:18 a.m.
386 views
Why Netflix ❤'s Scala for Machine Learning
Jeremy Smith & Aish, Netflix
June 12, 2019 · 12:15 p.m.
5025 views
Massively Parallel Distributed Scala Compilation... And You!
Stu Hood, Twitter
June 12, 2019 · 12:16 p.m.
958 views
Polymorphism in Scala
Petra Bierleutgeb
June 12, 2019 · 12:17 p.m.
1113 views
sbt core concepts
Eugene Yokota, Scala Team at Lightbend
June 12, 2019 · 12:18 p.m.
1655 views
Double your performance: Scala's missing optimizing compiler
Li Haoyi, author Ammonite, Mill, FastParse, uPickle, and many more.
June 12, 2019 · 2:30 p.m.
837 views
Making Our Future Better
Viktor Klang, Lightbend
June 12, 2019 · 2:31 p.m.
1682 views
Testing in the postapocalyptic future
Daniel Westheide, INNOQ
June 12, 2019 · 2:32 p.m.
498 views
Context Buddy: the tool that knows your code better than you
Krzysztof Romanowski, sphere.it conference
June 12, 2019 · 2:33 p.m.
393 views
The Shape(less) of Type Class Derivation in Scala 3
Miles Sabin, Underscore Consulting
June 12, 2019 · 3:30 p.m.
2321 views
Refactor all the things!
Daniela Sfregola, organizer of the London Scala User Group meetup
June 12, 2019 · 3:31 p.m.
514 views
Integrating Developer Experiences - Build Server Protocol
Justin Kaeser, IntelliJ Scala
June 12, 2019 · 3:32 p.m.
551 views
Managing an Akka Cluster on Kubernetes
Markus Jura, MOIA
June 12, 2019 · 3:33 p.m.
735 views
Serverless Scala - Functions as SuperDuperMicroServices
Josh Suereth, Donna Malayeri & James Ward, Author of Scala In Depth; Google ; Google
June 12, 2019 · 4:45 p.m.
936 views
How are we going to migrate to Scala 3.0, aka Dotty?
Lukas Rytz, Lightbend
June 12, 2019 · 4:46 p.m.
709 views
Concurrent programming in 2019: Akka, Monix or ZIO?
Adam Warski, co-founders of SoftwareMill
June 12, 2019 · 4:47 p.m.
1974 views
ScalaJS and Typescript: an unlikely romance
Jeremy Hughes, Lightbend
June 12, 2019 · 4:48 p.m.
1376 views
Pure Functional Database Programming‚ without JDBC
Rob Norris
June 12, 2019 · 5:45 p.m.
6374 views
Why you need to be reviewing open source code
Gris Cuevas Zambrano & Holden Karau, Google Cloud;
June 12, 2019 · 5:46 p.m.
484 views
Develop seamless web services with Mu
Oli Makhasoeva, 47 Degrees
June 12, 2019 · 5:47 p.m.
785 views
Implementing the Scala 2.13 collections
Stefan Zeiger, Lightbend
June 12, 2019 · 5:48 p.m.
810 views
Introduction to day 2
June 13, 2019 · 9:10 a.m.
250 views
Sustaining open source digital infrastructure
Bogdan Vasilescu, Assistant Professor at Carnegie Mellon University's School of Computer Science, USA
June 13, 2019 · 9:16 a.m.
374 views
Building a Better Scala Community
Kelley Robinson, Developer Evangelist at Twilio
June 13, 2019 · 10:15 a.m.
245 views
Run Scala Faster with GraalVM on any Platform
Vojin Jovanovic, Oracle
June 13, 2019 · 10:16 a.m.
1339 views
ScalaClean - full program static analysis at scale
Rory Graves
June 13, 2019 · 10:17 a.m.
463 views
Flare & Lantern: Accelerators for Spark and Deep Learning
Tiark Rompf, Assistant Professor at Purdue University
June 13, 2019 · 10:18 a.m.
380 views
Metaprogramming in Dotty
Nicolas Stucki, Ph.D. student at LAMP
June 13, 2019 · 11:15 a.m.
1250 views
Fast, Simple Concurrency with Scala Native
Richard Whaling, data engineer based in Chicago
June 13, 2019 · 11:16 a.m.
624 views
Pick your number type with Spire
Denis Rosset, postdoctoral researcher at Perimeter Institute
June 13, 2019 · 11:17 a.m.
245 views
Scala.js and WebAssembly, a tale of the dangers of the sea
Sébastien Doeraene, Executive director of the Scala Center
June 13, 2019 · 11:18 a.m.
661 views
Performance tuning Twitter services with Graal and ML
Chris Thalinger, Twitter
June 13, 2019 · 12:15 p.m.
2003 views
Supporting the Scala Ecosystem: Stories from the Line
Justin Pihony, Lightbend
June 13, 2019 · 12:16 p.m.
163 views
Compiling to preserve our privacy
Manohar Jonnalagedda and Jakob Odersky, Inpher
June 13, 2019 · 12:17 p.m.
301 views
Building Scala with Bazel
Natan Silnitsky, wix.com
June 13, 2019 · 12:18 p.m.
565 views
Diversity and Inclusion Panel: Bring the Thunder!
Panel
June 13, 2019 · 2:30 p.m.
244 views
Asynchronous streams in direct style with and without macros
Philipp Haller, KTH Royal Institute of Technology in Stockholm
June 13, 2019 · 3:45 p.m.
304 views
Interactive Computing with Jupyter and Almond
Sören Brunk, USU Software AG
June 13, 2019 · 3:46 p.m.
681 views
Scala best practices I wish someone'd told me about
Nicolas Rinaudo, CTO of Besedo
June 13, 2019 · 3:47 p.m.
2700 views
High performance Privacy By Design using Matryoshka & Spark
Wiem Zine El Abidine and Olivier Girardot, Scala Backend Developer at MOIA / co-founder of Lateral Thoughts
June 13, 2019 · 3:48 p.m.
753 views
Immutable Sequential Maps – Keeping order while hashed
Odd Möller
June 13, 2019 · 4:45 p.m.
276 views
All the fancy things flexible dependency management can do
Alexandre Archambault, engineer at the Scala Center
June 13, 2019 · 4:46 p.m.
389 views
ScalaWebTest - integration testing made easy
Dani Rey, Unic AG
June 13, 2019 · 4:47 p.m.
468 views
Mellite: An Integrated Development Environment for Sound
Hanns Holger Rutz, Institute of Electronic Music and Acoustics (IEM), Graz
June 13, 2019 · 4:48 p.m.
213 views
Closing panel
Panel
June 13, 2019 · 5:54 p.m.
400 views