Player is loading...
ScalaClean - full program static analysis at scale
Rory Graves
Thursday, June 13, 2019 · 10:17 a.m. · 43m 59s
This talk introduces ScalaClean which applies whole program static analysis to larger codebases. By combining data available from ScalaMeta and SemanticDB we are able to generate a full graph of all packages, classes, functions and variables within a codebase. We are able to use graph colouring to provide interesting analysis of the graph to infer dead code and correct visibility scopes. We can apply these changes back to the code reducing the size of the codebase and improves incremental compilation and IDE completions. This talk explores current progress and future direction of this work. Rory spends his life fighting to find a balance between writing ugly high performance code and beautiful maintainable code. He fights from the trenches to make software better, contributing to open source and has worked across the entire gamut of software from building a JVM to optimising grid computing. He believes that great tools are the key to unlocking the full potential of modern programming languages.
Embed
Transcriptions
Note: this content has been automatically generated.
00:00:00
come on
00:00:04
a few uh well hopefully this won't be too taxing on
00:00:09
the minds but hopefully interesting at the same time sort of
00:00:13
to that good what's colour cling which sort of source project um
00:00:16
i started back in january with another guy caught my scales um
00:00:22
doing a hopefully interesting things and have you discovering interesting things about my source code and about your source code
00:00:29
but i have a a one confession stock the stock to talk which is uh
00:00:36
if you've ever written a talking you submit an
00:00:38
abstract for conference you're like okay i've got five months
00:00:43
by the time i get to the in the five months is gonna be perfect it's gonna be ready i'm gonna run on everything
00:00:49
yeah it's not quite as truth or typed so uh we have tested it it does well cornell small
00:00:56
test project and of course will happen is we probably went along and try to work on the bigger project
00:01:01
and if you've ever done that it then fell apart in new and
00:01:05
fun and interesting ways we still try to debug a couple there is uh
00:01:09
new edge cases we found that does work but only some cases and we're gonna make it work for the other cases
00:01:19
so let's let's go back a step before i really start talking about
00:01:23
what we're doing and i wanna talk a little about our software quality
00:01:27
now the fun part that software quality as all of you know is we don't build statues
00:01:34
right we don't build pieces that go in the museum and a beautiful and a complete and up
00:01:42
we don't have this this is that we then had an extra floor to all some do such
00:01:47
a swimming pool on the top floor and some little the foundations are quite as good as they well
00:01:54
i said i look back at a later actually uses light across use
00:01:57
latex yeah so like to the region you released in nineteen eighty three
00:02:04
and there was a bug bounty on this but boundaries finding boxing like tech to know when the last release of late it was
00:02:12
you wanna make a guess
00:02:14
november two thousand eighty
00:02:18
so like tech you consider complete but it's it's evolved
00:02:22
and fixed a box so for is continually evolving process
00:02:27
and the problem is it is we talk about ten could at home we will
00:02:31
talk about the debt projects buildup but the thing about technical debt is the truth is
00:02:38
it continually accrues as would building software debts accrued
00:02:43
and it doesn't matter how good a program you well
00:02:47
right you will as you as you software changes involves bits
00:02:52
would die but so be added new books we introduced box uh
00:02:56
i've been there since nineteen eighty three will suddenly be found and suddenly be critical
00:03:04
one the chances measuring it's really hot how good your software if
00:03:09
you really is right i'm not using great magical software this perfect
00:03:15
i am
00:03:21
uh but my point is you know we'll build box right i turn
00:03:24
coffee into box and then to to more coffee hove into view box but
00:03:32
penny back is hot
00:03:35
as you go software keeping the noise down to double role he's hot
00:03:41
any slows you down with about technical data and and you can talk about think about it like mortgage interest
00:03:46
so you can either have a repayment mortgage your interest only interest only mortgages you basically
00:03:52
servicing the debt putting in twenty years time that that still that you still gonna keep high
00:03:58
and sometimes you compare bit more and you can reduce the data bit
00:04:03
but it's still evolve then as you had an extension you're gonna borrow some more from the bank i keep like it
00:04:09
and we keep doing this right we we had we do
00:04:12
this we keep asked softly good through processes code reviews automation
00:04:20
talk find is all those kind of things that should be kinda
00:04:26
it is important and
00:04:29
what martin fellas anything so and was really interesting to me is this is one of the first books i own about software quality which is
00:04:36
writing sorta code which was released in nineteen ninety three and actually if you go back and
00:04:42
read a is really dated now but a bunch of the stuff in the exactly really important still
00:04:49
to how do we keep coupling will consist design
00:04:54
we actively work on caring about what i lack of quality
00:04:59
my biggest advice if you take nothing else from this talk
00:05:04
don't reinvent the wheel
00:05:08
you might think that you know how to write a looking framework better
00:05:13
but you don't at some point you'll come back and go
00:05:18
okay now i'm gonna tell my own logging i'm gonna pay a lot for g. i. because
00:05:23
it was right on for long all i've walked into a company and what are they have they have their own to framework
00:05:32
was any good was it better than i care or not right so you've got to keep working uh as well you gotta keep evolving
00:05:42
your code key caring about keep re evaluating what you've done i come
00:05:46
back to code six might like to go off this stuff soulful right
00:05:52
you're welcome both the design and the housekeeping and i hear many people entering guy but surely there's
00:06:00
there's a school style and and more remover and scapegoats
00:06:04
and lint uh and it's gonna see mine is excellent
00:06:09
and intel decode inspections and could cover shows and
00:06:14
it's got six surely surely we've got this space covered
00:06:20
really yeah runs any of those automatically on the coda databases
00:06:26
okay i'm just gonna quit my talk down guard right well done every
00:06:31
a lot of projects i've come across worked on you come in and they've never wrong
00:06:37
i never thought about running any of that i i my chance you should if
00:06:41
you haven't run any one of these tools in your code go running on your code
00:06:46
and i can guarantee you might know agree with everything he says but you'll find at least one block
00:06:54
i challenge you and i i will buy you be uh if you
00:06:57
run these tools on your code and they don't find something major hum
00:07:03
yeah i'm gonna i'm not gonna i have lots of slides that people look at take care about looking at my slides later about what
00:07:09
these individual tools due on august skip through the entire lot but
00:07:14
there s. honours rolls up there's a lot of crossover during the tools
00:07:21
but they tend to check for local things
00:07:25
right it's very easy to check does does someone have the word now in that code
00:07:30
so they do today comparing to the string which is never going to hatch true
00:07:37
so they do unsafe things
00:07:41
and all those of their use and they they have suddenly cracked and you should run these cut if
00:07:47
you compare going into the string you're doing something wrong in your code is incorrect code and fix it up
00:07:55
you find it can be quite opinionated so if you don't agree with the programming style of the author
00:08:01
you're gonna find these tools flight got stuff that your okay
00:08:04
with for example i'm perfectly fine with of all inside a function
00:08:10
i know lots of people in the room might not feel that's the right answer but occasionally it is
00:08:19
as i say there will find things wrong with your code set
00:08:23
so what's wrong with all these tools well
00:08:27
they tend to work at the local information level
00:08:32
and there's a bunch of things i cannot tell you about the code
00:08:38
so for example who calls to who calls this method no intelligent
00:08:44
will tell you you can ask it and metals will tell you
00:08:49
but the little turtles tend to look at the direct data that's physically
00:08:54
that so it looks at the jaw file or looks cemented d. b. file
00:09:00
well who extends this um if you have to try to get that the language to tell you
00:09:05
what to the subclasses of this class you know why this is a problem because often it's just
00:09:12
to run time that i knows what classes that subclasses class you want data loaded at this point
00:09:19
but they don't is you know all the surplus across the problem a
00:09:25
is is when we talk about joss i'm only to what's meant to be
00:09:31
those piece of pie should contain forward references when you're running code when your
00:09:36
j. v. m. is running your code it doesn't care about who calls for
00:09:43
what it cares about is who flew calls
00:09:47
and the jar file effectively contains it says you know this method is defined by calling this method of doing this
00:09:54
and he would tell you exactly which classes i depends on but not who depends on it
00:10:03
that information doesn't exist in your job
00:10:09
you have to compute it is that when you open intelligently and you import projects the first time
00:10:15
that's exactly what he's doing when you're gonna be project may grant away for a bit it's effectively indexing
00:10:24
what was interesting about this is this is i. q. or croft problem
00:10:33
we can generate
00:10:36
these cross
00:10:38
information is there but effectively you have to take all the full references and to reverse index on
00:10:48
thing we can do interesting craft things
00:10:53
we can colour it and you can start to tell really interesting things about the program
00:11:00
so the simple cases for example is find uses its techniques now i have
00:11:06
not from from so i've gotta to bi directional prof i know who who calls it and what it calls
00:11:15
and it depends how rockies easy right
00:11:19
'cause i've looked at for references of addled placing the code to talk about cos i so i now have the list
00:11:26
what's this method override
00:11:29
you know what's the what's the hierarchy
00:11:32
you need to trace it back from the source and and generates information
00:11:38
one of the problems with graf things is when i surely examples like the graphics really trivial
00:11:45
right it's got enough for links
00:11:48
but even if i just look at the bill would grow off of a project suddenly it's it's
00:11:55
so big that i could no longer reason about what's the critical part in this in this depends
00:12:01
to this to build gruff turner no okay let's go let's go one step further let's look at
00:12:08
it a real program instead of just looking at the the build modules right walking you thought about this
00:12:18
so
00:12:20
these dependency graphs are huge feeling that your program to take
00:12:24
any any function in your program how many other functions physical
00:12:29
right i i i would suggest you find out factor is
00:12:33
at least five probably higher even for well written tiny little functions
00:12:40
so these cross get very big very fast so have we reason about
00:12:45
and then well we use programs reason about we don't do it ourselves
00:12:49
you you can't i can't i i work on projects a big enough
00:12:54
even medium size project i can no longer keep it all in my head
00:12:59
right i don't know i i assume everybody else has this problem as well but
00:13:03
want to program goes beyond a few files you you forget stuff and you rely on the
00:13:08
fact that you build in a certain why you can name in a certain mention to find things
00:13:14
so lots of time with plans easy catches fancy graph the whole project
00:13:20
it is practically c. s. face right it's just a list of references
00:13:25
and use that analysis do interesting things
00:13:31
there is a challenge this whole program analysis
00:13:35
which is you need to have a close for some shit
00:13:39
which is one and the lighting a program
00:13:43
you know if i want to say this method design used i need have all possible
00:13:48
colours that method within the scope of the thing i'm looking otherwise well i'm going to myself
00:13:56
which means you need to find the entry points now obviously the main method is easy
00:14:04
if you're using certain text frameworks you can use the annotations find them
00:14:09
if you if you're writing a library you going to have to tell
00:14:13
the tool where your entry points out what your public a. p. r. s.
00:14:18
actually i think as a generally useful
00:14:21
thing to do anyway things that a tool can't ultimately do is for
00:14:26
example followed something reflective like i can't see it right that is is magic
00:14:33
so if you're going to do that you're going to need to provide these tools extra information site this is an entry point
00:14:41
second week actually information well
00:14:44
we started using semantic d. b. has got a matter in fact i started writing this as a scholar fix rule
00:14:51
in fact attentions to scott fix was the first one goes over the coding does the analysis
00:14:55
and which has no result and the second one that applies the transformations because it's gotta fix
00:15:03
does one for the time it loads file it gives you the semantic information and this is what
00:15:08
you want to do with this file is that well i haven't yet got the back across so
00:15:12
what i do is i go through the first time for it to run run in nasa's role and
00:15:18
that spits out of a bunch files and then i run a modification rule which goes through and also logic
00:15:26
but you do end up having to use its colour selection and java reflection to get certain information out as well
00:15:33
there's there's a bunch of things that you have to to tie the knot i'm gonna i might come back to
00:15:39
that have time of it later um this probably more stuff that needs from a nice going to smell to d. b.
00:15:46
um to make this easier
00:15:50
so given all these kathy acts and all these one is what can we
00:15:53
do with it turns out i can do a bunch of really interesting stuff
00:16:00
so the first one to detect code detection
00:16:06
once at the entry points i can do simple graph colouring basically you go through the graph you colour the entry points
00:16:13
and every method that that those entry points touch you colour those new keep going until you until there's nothing left colour
00:16:22
and anything left them coloured is that like it's unreachable
00:16:27
funny enterprise you program it cannot be executed by your program
00:16:33
great
00:16:35
one of the fun ones i thought was just a a nice little side effect is turned out i
00:16:39
think to be quite important is pretty cisco i mean go through these in more detail in a bit
00:16:46
the take your program it's collect he's topic by default
00:16:52
uh_huh and if you uh so danielle us talk one of her things was you
00:16:57
should define europeans if you're gonna writer class you should define what you publicly guys
00:17:07
but the fact is what you get a certain stars like you need to then go through each month ago who causes is it public is a project
00:17:15
it's a pain and you can do this automatically and actually really trivial once you got back links this is trivial i i can i
00:17:22
can look at a method and go uh the only people chorus ringside
00:17:27
this package are connected packs profit or it's i can sculpt it perfectly
00:17:34
and when the other ones looking at the stewards talk yesterday is built on it
00:17:42
you find you build dependency you you've dependency graph until it's tends to have we patterns it
00:17:49
um i'll come back to exactly what uh build i'm does like to 'cause it's it's easy to scrub the diagram
00:17:54
and there's a bunch of the cleanups and and i'm sure that once this is make sure they'll be a
00:17:59
bunch of stuff that i haven't even thought of the people go oh yeah i can do this i can detect
00:18:04
that diego if using the room actually came up with something and raise the
00:18:07
ticket before the software with even ready to run which is right so let's uh
00:18:14
let's have a look at the code analysis yeah finding the code is hard to do
00:18:19
manually it's intended it will tell you if you got a project method which some code
00:18:26
right as we said most my method on private because i haven't got around to doing the uh the scoping thing
00:18:35
yeah
00:18:36
but the problem is when you're looking at code i've i've spent days trying to re factor piece code
00:18:42
i'm sure i'm not the only person the room trying to fix his code days nine days re faction fiddling trying to get the
00:18:49
new i. p. i. too but with the l. d. a. p. i. to discuss well after two days the the entire chunk can detect
00:18:58
right it's just add up
00:19:02
yeah a trust me if if you're new to the industry at some point you
00:19:07
have this problem you'll come across curly like these are and it dries you slowly bonkers
00:19:14
but also intended to tell you this is cool you know if it was cool but
00:19:17
i'll dark over something else and and you can't immediately see maybe you can't midi c.
00:19:24
but the code is that you know it sometime distributes all this
00:19:28
is matthew design used to like to write cock right but often not
00:19:35
a circular references or others is that you can't you can't see it by a simple inspection
00:19:41
and those can be huge chains methods or huge some crawls
00:19:46
i have a personal belief that if you take any large enough code base
00:19:52
if i find conserved i'm gonna say twenty percent of the code
00:19:55
is dead especially if the code evolved over over a ten year project
00:20:02
i think i personnel one run this on some really loud cry bases and try and try and prove this but i
00:20:06
think i think it's at least twenty percent and actually i lean towards thirty percent of any any large databases just add code
00:20:14
i feel pretty that uh soon hopefully it's easy impure functional code because you can't have functions that you know
00:20:24
there's not the you don't have the dependencies you don't have instance instantiation all that stuff it's it's a lot easier
00:20:31
please it is simply a graph colouring problem adjusted you colour financial points
00:20:37
and possibly the test and and one interesting things you can do the analysis as well
00:20:42
this code is touched by test code does not to not reachable by by the the main caught you
00:20:48
know that's an interesting as it's it's great having test but are you testing something that you actually care about
00:20:57
anything that is not coloured is dead by definition
00:21:02
and so you can compare which reflects a few quick sample so here's
00:21:06
my like trivial coloured suit colour you start within should points you mark values
00:21:13
you can go through any of those and you say the my method calls bar
00:21:19
the bar is life and we can easily detect no of see
00:21:22
this the trivial slide example but it doesn't matter how big the graphics
00:21:28
you which right covering the graph until it's until you can colour them all
00:21:32
and it working it'll work on cycles and it'll work on all the other parts
00:21:40
but there are some limitations it can be as i said really difficult to detect the entry points
00:21:45
if you've got specially state guy loading your code you are gonna have to
00:21:48
market or had logic to the tool to tell you what the inch point is
00:21:55
if use reflection aggressively this is not going to work you again you're gonna have to provide hints
00:22:04
um and the solution is simple right you annotate all you had come thick
00:22:12
but
00:22:14
it's actually really easy to do
00:22:16
and i think it's gonna provide some really interesting and and the funny part is
00:22:19
what i then did is my scholar six rule went and deleted the unreachable code
00:22:25
and the beauty is it's really funny because you run the tall and like i give your code base and the guarantee is
00:22:31
this code base must compile before and otherwise i all bets are
00:22:35
off right and it must compile afterwards and still pass the tests
00:22:40
and he does that i've won right i've i've reduced yoko complexity for free
00:22:48
so it can be difficult to jet engine points is that is that sounds like it
00:22:54
so privatisation
00:23:00
now this is the the second use case that i've been playing with and the the only cure simple
00:23:07
reduce the scope of everything to the minimum it's allowed to pay to make your program still function
00:23:13
again if i run this rule what i expect suspect you
00:23:16
programmed column full hand unexpected compile run afterwards with no visible differences
00:23:24
in fact i either it will kampala one compiler afterwards because what would you do this while it's compactness
00:23:35
right minimising the scope of things is good few program is good few developers is good fewer
00:23:41
uses if you're alive your for you know having everything public is just is just an empty packet
00:23:48
is that the actually really helps code completion this is one of the use case i think actually where
00:23:55
this is gonna come out to be really useful for logical basis house incremental compilation you could compel faster
00:24:05
and uh if you so all uh students talk yesterday um
00:24:10
about part one compilation i help soft as well because housing violent typing
00:24:17
so here's another trivial program
00:24:21
um
00:24:23
how does privatisation opus it hides methods that we don't care about
00:24:26
so if i just have public methods and i use you know
00:24:32
suggest code you know dot control space it suggests all three methods
00:24:39
and the other is actually it's just methods have practically about methods i don't care about
00:24:45
it's like the appropriate scoping on these
00:24:49
intended to does a better job of suggesting what the reckon patients
00:24:54
right this make your life or my life easier
00:24:58
because i i don't know about you but eyes open code was imported a
00:25:02
library or something that i've gone what message does it have an if she
00:25:10
she
00:25:13
remote compilation well it which uses the dependencies between
00:25:16
files so zinc asked to decide what to compile
00:25:23
like if you probably k. p. i. of your class changes at all it goes who depends
00:25:28
on this file because has gone compel those as well to reduce your scripts in right guys
00:25:34
means in today's few files
00:25:37
so if you are only changing the price implementation of a method of the class
00:25:42
and you haven't changed public a. p. i. thing can be done much smarter about it and get away with only compiling that cops
00:25:50
right so you're helping the compiler detect things and make your life better
00:25:59
so the last i'll some may is one of the the sort of code not it's
00:26:03
it is we can make compilation in general fast by doing the stuff i'm at my typing
00:26:12
so i mean it describes into more detail but what you want to do what you can do if you've got modular module b.
00:26:19
what happens at the moment is you compile module a you get a job
00:26:23
and you feed that into module b. writes your dependencies effectively uh uh the bonneville
00:26:30
but actually most of the time most the cases what you actually need to compile more to be
00:26:38
he simply the public a. p. out much alike you need the types and
00:26:42
you need the classes but you don't actually need the internal implementations of those method
00:26:49
in fact a stew was talking yesterday about the google do this for for job is a job at all
00:26:56
it's of cheats it generates an an on the job which the interface towards fetch clear
00:27:00
the minimum job you can generate with just effectively anti methods and the outlines of the classes
00:27:07
because that's enough for the next stage compile not to run but to compile
00:27:14
and of the privatisation he's a great stepping minimal in working out
00:27:19
what that probably keep guys so let's say that let's walk through this
00:27:24
so he is my is my simple bills i'm not gonna timelines because you can imagine that
00:27:29
okay so i have many many calls on my machine even my really old laptop now
00:27:37
and my compiler incentives right
00:27:41
so let's let's be smart it might hit my two modules and the compiled one
00:27:45
off the up but in fact devices start to break down the compilation i'm gonna
00:27:51
should we break into three phases which is tossing files type checking everything and then generating jobs
00:27:58
and and and doing all the other housekeeping well actually i could start by doing the processing talk
00:28:08
there's nothing stopping me part doing the pausing simultaneously
00:28:14
but once we start using out when typing to do the work in fact you can do better
00:28:22
so i can pause at the same time
00:28:26
but module to only depends on the result of that one topping
00:28:31
module one so i can now stop our allies in them much earlier
00:28:38
a module three so now i'm doing a lot more working assume you have got enough calls
00:28:44
and if that one of the interesting things is i can buy calls off high muzzle like i can i
00:28:50
can start one second start lifting the stuff you can have as many cause i need for short periods of time
00:28:59
one of the key things is actually causing doesn't take full
00:29:03
what makes the what you wanna do then is minimise the amount of time to get the end of that one type thing
00:29:11
right that's ease your critical half in compiling
00:29:16
that all the other stuff is easy ease noise right because if i've got
00:29:19
enough modules that all they are gonna overlap and i'm gonna start running out calls
00:29:26
how we present when typing
00:29:28
well i what what what what do i need to provide the compiler
00:29:33
to do outline talk well in eastern though the public methods and the types
00:29:39
so i can do two things right number one is i can minimise the number public methods
00:29:45
and number two working going runs colour fiction say all those public methods now describe the types
00:29:51
so i've now got a really small was a six
00:29:55
this really easy to compile
00:30:00
and then you go one step further which is been alluded to
00:30:04
a couple times once i've got the one types within a single compile
00:30:11
all of the files are effectively independent
00:30:16
right because compile
00:30:18
class b. i i only need the public information that prosody
00:30:25
and vice that so in fact talking up our allies
00:30:31
i love to type check and ten sheets or the bit further down as well
00:30:36
i'm not gonna draw too complex diagrams but my point is severely
00:30:41
began to go for this class here is a public a. p. i.
00:30:45
here's all the knowledge you here it's a knowledge anybody else
00:30:47
needs to depend against it i can now speedup compilation hugely
00:30:56
and is it's it's a really interesting aaron this whole bunch of work going on um chases
00:31:01
all said it's an interesting fun compiler flags says a comedy exact my sex outline type something
00:31:09
mine is why outline discover see as as a
00:31:12
prototype of of doing exactly this directly within scholars say
00:31:19
anything we can do to minimise mention for you and what it would do is if it need to type if you
00:31:24
haven't ascribe to talk to public method he will come into
00:31:27
as much complication is it needs to work out the outline
00:31:32
which means if you code is completely on described i completely public it's basic gonna
00:31:38
have to do the entire thing other than me in size of the methods anyway
00:31:43
so anything we can do to help it's going to make our lives easier
00:31:47
once you get that so each module is no compelling really fast but i end up with another problem
00:31:54
which is of like to build diamonds if you look at large trace of the module dependencies
00:32:01
last code base you find you start with often you have some uh you till package
00:32:07
right to the bottom some really interesting things which is might be really fast
00:32:11
compile but everything depends on its gonna guess first then i'm gonna have mccall packages
00:32:17
right and that great 'cause i've i've separated out nicely they can all become pouting parallel
00:32:23
and then i end up with some call it pushed depends on all of those
00:32:30
right so it becomes a button that right okay i compare as a step really well and then they
00:32:35
end up with one or two modules which suddenly i'm i'm now i can't do anything downstream until that's done
00:32:44
and then i might get a branch out again and then i've got my my completion but this to build element
00:32:50
right my my dependencies on the on the end of stock behind the single module in the middle
00:32:57
right which really limits my parents they should if i can move stuff around then i can do a lot more work earlier
00:33:08
no
00:33:10
and we get so i'd admissions split i've realised that there's there's some dependency and effectively with the croft again
00:33:18
the cross knows which modules the code is in
00:33:21
you can work out which things depend which effectively within
00:33:27
i give a module there might be some cross there might be two independent graphs
00:33:33
scott and i can start separate into
00:33:38
or i can i can suggest to the to the code all for this this module can broken it to
00:33:43
how much i break ins to i can call those empower oh i have
00:33:45
a much better dependency graph but it's a great picture but you get the idea
00:33:51
i can detect you know should should a message to the class peopled
00:33:56
back into early module for dinner later model 'cause nobody other
00:34:01
than the model the index because about it working part later
00:34:05
you know you can start to juggle your code and i think you could even
00:34:10
automatically jiggle the code to provide better build structures to optimise the bill pipeline
00:34:21
so those are the the the the key thing so i've been i've been playing with
00:34:26
in terms of the fundamental ideas that the two key things i think a most interesting
00:34:32
in this sort of all our privatisation your minimises scope and that card novices
00:34:39
'cause i think those will provide
00:34:42
to write benefit to any any and all code office but in fact if you go back to the
00:34:50
the things we look at earlier with you discover stall it's gotta fix lint
00:34:55
uh there's a bunch of other things you can detect any further about code
00:35:02
and this leads to a whole bunch of other potential small
00:35:06
you know my crotch migrations and and and tweaks that we can
00:35:12
remove unused from such a as in code where you've got a
00:35:14
parameter threaded through twenty methods and at some point it became unused
00:35:22
but it's but you did not
00:35:25
right so you're still trying to fiddle through the what what what value twenty to pasta
00:35:29
try working through all this code and it turns out that parameter is never actually used anymore
00:35:35
once you got cleverness is you can actually on friday you because uh this parameter is a
00:35:40
bit like an amusement that this parameter is unused old all this parameter is always seven always true
00:35:51
so you can request approaches
00:35:54
and you can relax types
00:35:57
so one of the fun ones and and actually uh diego uh suggested this as i say
00:36:02
as a as a as a feature request on the colour based for the cowboys was even working
00:36:08
which is if you've got a method that takes up the
00:36:14
basis of possibly but you only have a call a method
00:36:20
then you can replace the the the baby with that
00:36:24
i knew why didn't you know you've like the types in you've you've you've come pouts the interface not the implementation potentially
00:36:33
so
00:36:38
so this is a fun
00:36:40
so what i've learned on this journey so far
00:36:44
actually i let some of this room for men's on as well
00:36:46
which is tooling and i'm sure i'll ever agree tolling is really fiddly
00:36:52
you find all the shop ages in types and jobs and representations
00:36:59
yeah the the fact is i ended up with a semantic d.
00:37:02
b. symbol discover c. symbol your and a and a java reflection symbol
00:37:09
and there will be i know that the signs saying that because the original
00:37:13
the other things it's really actually hard to tell all proof it so i'm still
00:37:21
working out how to deal with some research cases but that's all the interesting part is that's all in the analysis phase
00:37:27
it's all about putting out information about what this code mates and there's various ways of doing this you could
00:37:33
go back to the compiler plug you cook about kampala club because you have a lot
00:37:36
more information in one form at that point you could add more to ninety d. b.
00:37:42
yeah depending on whether information should be there's a bunch of stuff
00:37:45
to do but it it turns out sometimes this stuff is just fiddly
00:37:51
and invest don't don't down it down of the bells knows that you could you've got a job with
00:37:55
java jar undiscovered 'cause program and the types sort of
00:37:59
mature and for example one of the things i found
00:38:04
with a skull matters gothic is i couldn't tell
00:38:10
when i've got method uh a method overriding apparent method which one it
00:38:14
over rights and you've got to be similar nineties actually really hard to tell
00:38:20
exactly the the exact hierarchy and and it's not the the information doesn't exist in some entity be at this point
00:38:28
and so as as it reverse engineering this information is just really painful
00:38:34
so what next
00:38:37
well actually development getting more data might my goal my next
00:38:42
goal is to make this work on the aqua code base
00:38:47
as i said the first time i tried running attack a blew up in new and interesting
00:38:50
ways and still working through that but hey that's that's real programming right what's in the tests
00:38:58
and then testing on even bigger bases um the community build for example be funk thing to run this on
00:39:06
and seeing whether it leads i think once you have the the information i think there's a bunch of really interesting stuff to do
00:39:13
and there's a whole program static analysis i think the really interesting
00:39:17
space i think is a whole bunch of things we can detect indeterminate
00:39:22
and if uh to make our lives easier both from a us of development point of view a comment his point of view and just
00:39:30
and all those work is not a turnkey solution yet the goal is for it
00:39:34
to be just another tool you run a promise wanna get their outreach about it
00:39:41
there's lots things to
00:39:43
so with that i am i wanna finish with with a quote which is um the founder vince
00:39:50
crisis offered like gardening wonder go behind the shed cleanup if nobody ever goes there doesn't matter a lot
00:39:59
i think that's a false analogy i think it's closer to being i'm living in a house was time renovating
00:40:05
it and that there's a few things which bite me every single day when the plumbing doesn't quite work right
00:40:11
i think you need to go and fix those things i think very much
00:40:23
so we have a few minutes of questions or
00:40:28
thank you very much for talking was really interesting um question about
00:40:32
um that couldn't be that could analysis what you said that fall
00:40:37
reflection all for probably came points they needed to be annotated you know the to convey
00:40:41
does collect clean but this should not be considered that code m. ah have it
00:40:46
is d. of doing that was pull guards and other tools and that was it
00:40:50
not a nightmare to get it right do you did you find ways to to
00:40:53
to work around that i'm all sort of offer the the bits i played with
00:40:58
uh i've just for example i that had a a c. s.
00:41:01
v. file which is like here all the classes which are considered
00:41:05
entry points so if i have an interface which i passed other people i just lists that as a you cannot touch this class
00:41:13
okay uh and treat this as an entry point all using a
00:41:16
an annotation to say this is an entry point and i'm responsible for
00:41:20
and so far this has been enough and you this this whole thing when i'm not so so far this is been enough okay thank you
00:41:32
interesting and yeah when you mention the rebar or uh yes i
00:41:37
will i will treated after the the talk and it's in my
00:41:41
yeah i i will just basic it in my it's
00:41:46
a cab dot com slash lori grey slash colour clean
00:41:51
um but i will actually put on slide
00:41:59
but the questions
00:42:01
and uh
00:42:05
i guess i guess i guess i i guess i i i i took the um
00:42:10
um the the sort of the close world assumptions as as mean as replied to
00:42:15
like complete applications as letters but then in one of the last
00:42:18
slide design you you can apply this cough how are you gonna bounce
00:42:23
um uh that's hitting and she points something as large not selected as as as a uh as a
00:42:30
library and all this and thank you my at my in that this is in oz thank you all
00:42:35
grateful but you know how to lie outside shapeless the
00:42:38
what other borrowers entry points has now uh the thing is
00:42:42
for for library like shapeless i suspect there are there
00:42:45
are parts which are things you expect people to call on
00:42:49
and those are gonna be your your your public a. p. i. to
00:42:51
all the uses and a bunch of code which is into an implementation detail
00:42:57
and i think you end up having to i the list to annotate what those those public
00:43:03
tosses or was probably packages and effect for you say these are things you know about
00:43:09
these uh these are dead back as you say it's a bit harder because people do extent
00:43:16
but you need to it so you need to um annotate correct
00:43:21
'cause is an entry point which people extend to bite back out
00:43:27
but i suspect when you when you look at it there's a there's a few
00:43:30
classes which are entries or extension points and a lot of classes which are implementation detail
00:43:37
and i think it will do i i i accept yeah i agree for for a light breeze
00:43:42
this is more complicated fall applications is it easier um and that's something yeah it's something still open
Conference Program
A Tour of Scala 3
Martin Odersky, Professor EPFL, Co-founder Lightbend
June 11, 2019 · 5:15 p.m.
8337 views
A story of unification: from Apache Spark to MLflow
Reynold Xin, Databricks
June 12, 2019 · 9:15 a.m.
1267 views
In Types We Trust
Bill Venners, Artima, Inc
June 12, 2019 · 10:15 a.m.
1569 views
Creating Native iOS and Android Apps in Scala without tears
Zahari Dichev, Bullet.io
June 12, 2019 · 10:16 a.m.
2232 views
Techniques for Teaching Scala
Noel Welsh, Inner Product and Underscore
June 12, 2019 · 10:17 a.m.
1296 views
Future-proofing Scala: the TASTY intermediate representation
Guillaume Martres, student at EPFL
June 12, 2019 · 10:18 a.m.
1157 views
Metals: rich code editing for Scala in VS Code, Vim, Emacs and beyond
Ólafur Páll Geirsson, Scala Center
June 12, 2019 · 11:15 a.m.
4695 views
Akka Streams to the Extreme
Heiko Seeberger, independent consultant
June 12, 2019 · 11:16 a.m.
1552 views
Scala First: Lessons from 3 student generations
Bjorn Regnell, Lund Univ., Sweden.
June 12, 2019 · 11:17 a.m.
577 views
Cellular Automata: How to become an artist with a few lines
Maciej Gorywoda, Wire, Berlin
June 12, 2019 · 11:18 a.m.
386 views
Why Netflix ❤'s Scala for Machine Learning
Jeremy Smith & Aish, Netflix
June 12, 2019 · 12:15 p.m.
5027 views
Massively Parallel Distributed Scala Compilation... And You!
Stu Hood, Twitter
June 12, 2019 · 12:16 p.m.
958 views
Polymorphism in Scala
Petra Bierleutgeb
June 12, 2019 · 12:17 p.m.
1113 views
sbt core concepts
Eugene Yokota, Scala Team at Lightbend
June 12, 2019 · 12:18 p.m.
1656 views
Double your performance: Scala's missing optimizing compiler
Li Haoyi, author Ammonite, Mill, FastParse, uPickle, and many more.
June 12, 2019 · 2:30 p.m.
837 views
Making Our Future Better
Viktor Klang, Lightbend
June 12, 2019 · 2:31 p.m.
1682 views
Testing in the postapocalyptic future
Daniel Westheide, INNOQ
June 12, 2019 · 2:32 p.m.
498 views
Context Buddy: the tool that knows your code better than you
Krzysztof Romanowski, sphere.it conference
June 12, 2019 · 2:33 p.m.
394 views
The Shape(less) of Type Class Derivation in Scala 3
Miles Sabin, Underscore Consulting
June 12, 2019 · 3:30 p.m.
2321 views
Refactor all the things!
Daniela Sfregola, organizer of the London Scala User Group meetup
June 12, 2019 · 3:31 p.m.
514 views
Integrating Developer Experiences - Build Server Protocol
Justin Kaeser, IntelliJ Scala
June 12, 2019 · 3:32 p.m.
551 views
Managing an Akka Cluster on Kubernetes
Markus Jura, MOIA
June 12, 2019 · 3:33 p.m.
735 views
Serverless Scala - Functions as SuperDuperMicroServices
Josh Suereth, Donna Malayeri & James Ward, Author of Scala In Depth; Google ; Google
June 12, 2019 · 4:45 p.m.
936 views
How are we going to migrate to Scala 3.0, aka Dotty?
Lukas Rytz, Lightbend
June 12, 2019 · 4:46 p.m.
709 views
Concurrent programming in 2019: Akka, Monix or ZIO?
Adam Warski, co-founders of SoftwareMill
June 12, 2019 · 4:47 p.m.
1974 views
ScalaJS and Typescript: an unlikely romance
Jeremy Hughes, Lightbend
June 12, 2019 · 4:48 p.m.
1377 views
Pure Functional Database Programming‚ without JDBC
Rob Norris
June 12, 2019 · 5:45 p.m.
6375 views
Why you need to be reviewing open source code
Gris Cuevas Zambrano & Holden Karau, Google Cloud;
June 12, 2019 · 5:46 p.m.
484 views
Develop seamless web services with Mu
Oli Makhasoeva, 47 Degrees
June 12, 2019 · 5:47 p.m.
785 views
Implementing the Scala 2.13 collections
Stefan Zeiger, Lightbend
June 12, 2019 · 5:48 p.m.
811 views
Introduction to day 2
June 13, 2019 · 9:10 a.m.
250 views
Sustaining open source digital infrastructure
Bogdan Vasilescu, Assistant Professor at Carnegie Mellon University's School of Computer Science, USA
June 13, 2019 · 9:16 a.m.
375 views
Building a Better Scala Community
Kelley Robinson, Developer Evangelist at Twilio
June 13, 2019 · 10:15 a.m.
245 views
Run Scala Faster with GraalVM on any Platform
Vojin Jovanovic, Oracle
June 13, 2019 · 10:16 a.m.
1342 views
ScalaClean - full program static analysis at scale
Rory Graves
June 13, 2019 · 10:17 a.m.
463 views
Flare & Lantern: Accelerators for Spark and Deep Learning
Tiark Rompf, Assistant Professor at Purdue University
June 13, 2019 · 10:18 a.m.
380 views
Metaprogramming in Dotty
Nicolas Stucki, Ph.D. student at LAMP
June 13, 2019 · 11:15 a.m.
1250 views
Fast, Simple Concurrency with Scala Native
Richard Whaling, data engineer based in Chicago
June 13, 2019 · 11:16 a.m.
624 views
Pick your number type with Spire
Denis Rosset, postdoctoral researcher at Perimeter Institute
June 13, 2019 · 11:17 a.m.
245 views
Scala.js and WebAssembly, a tale of the dangers of the sea
Sébastien Doeraene, Executive director of the Scala Center
June 13, 2019 · 11:18 a.m.
661 views
Performance tuning Twitter services with Graal and ML
Chris Thalinger, Twitter
June 13, 2019 · 12:15 p.m.
2003 views
Supporting the Scala Ecosystem: Stories from the Line
Justin Pihony, Lightbend
June 13, 2019 · 12:16 p.m.
163 views
Compiling to preserve our privacy
Manohar Jonnalagedda and Jakob Odersky, Inpher
June 13, 2019 · 12:17 p.m.
302 views
Building Scala with Bazel
Natan Silnitsky, wix.com
June 13, 2019 · 12:18 p.m.
565 views
Diversity and Inclusion Panel: Bring the Thunder!
Panel
June 13, 2019 · 2:30 p.m.
245 views
Asynchronous streams in direct style with and without macros
Philipp Haller, KTH Royal Institute of Technology in Stockholm
June 13, 2019 · 3:45 p.m.
304 views
Interactive Computing with Jupyter and Almond
Sören Brunk, USU Software AG
June 13, 2019 · 3:46 p.m.
681 views
Scala best practices I wish someone'd told me about
Nicolas Rinaudo, CTO of Besedo
June 13, 2019 · 3:47 p.m.
2708 views
High performance Privacy By Design using Matryoshka & Spark
Wiem Zine El Abidine and Olivier Girardot, Scala Backend Developer at MOIA / co-founder of Lateral Thoughts
June 13, 2019 · 3:48 p.m.
754 views
Immutable Sequential Maps – Keeping order while hashed
Odd Möller
June 13, 2019 · 4:45 p.m.
277 views
All the fancy things flexible dependency management can do
Alexandre Archambault, engineer at the Scala Center
June 13, 2019 · 4:46 p.m.
389 views
ScalaWebTest - integration testing made easy
Dani Rey, Unic AG
June 13, 2019 · 4:47 p.m.
468 views
Mellite: An Integrated Development Environment for Sound
Hanns Holger Rutz, Institute of Electronic Music and Acoustics (IEM), Graz
June 13, 2019 · 4:48 p.m.
213 views
Closing panel
Panel
June 13, 2019 · 5:54 p.m.
400 views
Recommended talks
Formal verification of Scala programs with Stainless
June 14, 2019 · 2:20 p.m.
118 views