Embed code
i think we're much ah let me start by thanking the previous speaker thing we sure are a lot of what you just said
to our quest to make our research more useful so everything that just that is really what we feel so
alright so
but i'm trying to go back to the to the roots of the the the fusion of repairs were research so basically paper
but not just the paper also the code the data and instructions to apply this code of the data
and uh be able to redraw the results that you had on the paper okay very simple idea
well let's take a step back and why would you do this as as the
previous because sad i mean there's a lot of papers coming out and
one important thing to realise is that we're leaving more more in experimental wage
okay the time where everything was hard to prove the masses kind of
more more over and work to see is that there's a lot of large
data sets and people were pushing a lot of experimental evidence nowadays okay
um so the conclusion of that is that it becomes harder to reproduce
the exact conditions all the claims of a given paper okay
um if you don't have the actual code okay if it's not an exact paper about
a mess in the mathematical method or it's not conception is simple enough to reproduce
and to be frank i mean who wants to go actually on and months of work and try to reprise of people like
that it's it's really tough you you know in here you basically have the mass you have some figures but you
there's no space for the other for example describe mathematical tricks that actually
took to actually be able to to completely implement isn't software
okay so we do it very often to to boost the impact of our
research okay so this means to increase our possibility of of usability
um this was pushed back to two thousand twelve but i'm sure it's true it's true true okay
um basically if you look at the the current number off of repairs
were such papers it's still very very small so there's an advantage
now doing that is because i can stand out in front of this very a big mess of papers that we have okay
only ten percent off tape this transactions in in image processing
provide source code and we know from it that
basically just provide a source code is not making it necessary reproduced will as it was highlighted before
and a statistic your work is also more valuable if it is reproduced were
senator k. thirteen out of the fifty most i think they present
in in company which is very important um um journal in
computer science provided the source code okay and the average
number of citations for people to do provide source code in in tape is sevenfold of papers that don't
this also here which is in the in the footnote um the find
six different levels of rivers ability and of course what we're trying to do here is to make things reproduce bowl
and easy to reproduce for people okay so that should take a very small
amount of the time i think the time here is not very important
but it if it takes a day of work it's fine okay and the idea is basically that we try to figure out solutions that work
for free okay that you don't have to pay for so get out of the
loop where you have models that are viewed on proprietary software okay but
if you look at it's there is there are elements which actually missing from this okay
the question the first question and i think this was already pointed out the fall
is how much software you actually need to make an experiment reproduced okay
do you need the whole of matlab to be installed only on platform we need it
stops being the the mass library to be available they need to be observed version
if you're using high some you know you'd is versions of the the the levers that you depend on the date weekend
um let pack and so what we actually need to complete all asks what about the machine do you
also need to you know make sure that the machine is available for the present reproduce it okay
now this for example uh we're talking more more more more about keep you process important we needed you machine to be
okay another important and all i think we can uses on that definition is
how much time do actually need to keep it reproduced will for okay
is it one day so you publish the paper and next day you know the thing
is not reproduce by anymore i think that's not really the anvil right so
you're thinking here more about a few we use to want to keep your paper
visible for for at least a few years after after uh it's been published
now if you think back on how many papers actually be a month painting at a certain point in time
if your student and you publish one or two papers but here and you're
getting a few papers a few years of papers that means for five
which i think is quite doable for a single person but if you look to
oppose doc it's more like ten abruptly there's more like thirty or fifty
and if your little to multiple groups you're talking about hundreds of paper that actually meant painting here so few years okay
you've actually talk about very long term mountains numbers numbers would explode of course
know how to organise this okay you can leave each by on itself and that's basically what is most done yeah
you research to figure out independently every student every post
talk to whatever they they think it's reasonable okay
of course this makes the work more independent from each other okay but
the work becomes very difficult to carry on somebody to pine okay
and uh this includes of course to compare results even sometimes within your same group okay
other boats as others bus reaching all the parties okay only one
you don't know because i'm not holding yourself this question okay
and of course you're constantly reinventing do we we basically have no reviews across the across the
the set of people your your your working us okay and the second option which we think is is better is to
actually put in place a set of tools and procedures that allow workers ability more to take place naturally within the
within your products and collaborators okay so that is that's we move into another angle in where
work is actually done together okay the results up here on the same grounds okay
and everyone is basically testing the same code base okay so this means of course less butts
people looking more eyeballs looking at the core would okay and if a fixed
is required somewhere that everybody's going to profit from that fix okay
there's a lot of softer use that's comes you spontaneous and there's an advantage
is that people because you have so just from softer community around your
your your area of work that new colleagues can basically start really
immediately to produce very high quality results okay but you trust
now if you try to do that you're gonna be face was a lot of questions okay first the very simple
one is how can i make sure that people can still download my contribution where should i place it in
the the second on which is very important is any point of time ah you made us contribution does is to work the same
okay and if it doesn't what have i done to break it was i was it somebody else or some software the panel
how do i fix it and how he redistribute that takes okay is the problem on
my site for somebody to pack my stuff but they do actually tell somebody
that they should fix the softer okay and this maybe mask works of using matlab
looking it does happen we do have a case but when that happened okay
so what does it take okay if you talk about i'm hopeless instructions which is
one part of rubber disability appear gonna need a website to host it
talking about availability here of your they have to think about long term availability of your of your softer
you need of course to create casts a that is going to help you answer the question does it still
work the saying if you don't make cast then there's no way you can answer that's simple question okay
and of course you need revision control system that going to answer the question what have i done to break it
okay i didn't have to figure out every distribution mechanism for fixes and improvements all the time
okay now there's the other angle of it which is the data itself of course you also have to think about a place
to host it and of course to think about users protocols in gonna be testing are you your making experiments was
of course redistribution taken for fixes you also patch data line you pension protocols you
make mistakes on your data collection so you should be entitled to fixed
we're not covering this data part you think there is a lot of stuff that has already been done i think we're doing it quite
a good job but the up as well already we do have a big border where we put most of our data sets available
i'm gonna talk about the first five he okay and introduce you something that we we we know
we've worked for the last five years well we just call bob okay what is bob
so ball is in a word a framework for reversible research okay
in short it's an open source to get so everything is free of charge okay
for signal processing machine learning specifically okay that's putting solutions for almost
all of the of all the problems that i've just mentioned
obviously you're all for it it's written on the mix of licensee plus plus it has a very permissive licensing scheme okay
it's one thing that idea whatever get let instance that whole school there which is very nice because
you can get cool display you should trackers you know computers integration in the planet as well
the packages i hosted on a public price and um um
package index which is um very available and there's lots of the huge community around it but not things that
it is a lot of unit integration test that we actually do it every come eat
and every piece of software we have most of the pieces of soft we have
okay and the check as well every every every every day once a day
if everything actually works to be still okay it doesn't break anything else on that sometimes people make changes and
that break some of your looking and this may mean results on your paper that you just which
and we also provide a mechanism than to the point everything so the whole softer stack using on
okay an overview so basically in this area of research most problem can be surprise surprise this
falls so you have a database way making castle experimental computer science on that to change
that helps in solving your problem creating a model that allows you
to analyse okay so the three three basic construction box okay
the database provides some sort of an abstract problematic access to the raw
data so basically the basic io the lousy to read data from
the disk okay also transformed data so the features that you want to
temporarily store a while you're processing dating site your to change
and as well the use of protocols how to actually use that data to reproduce the experiment
i have um well it's been a bit bigger okay i'm the user protocol is the
is the is the the problematic in entity that actually trying to experimentation okay
the to change what is the technique to solve that particular task okay typically will contain some preprocessing feature extraction machine
learning and so on so forth and the allies is basically set of tools that allow you to summarise
into single speller although deportment result and great great figures okay
so bob is actually sat very large set of components okay
that i haven't uniform well as we try to make
it as uniform as possible sorry test set of interfaces the follow basic construction units okay so we have
elements that help us to create such a database interfaces elements
where y'all signal processing machine learning and metrics of course
okay our basic honed install also you gonna get a lot of packages that
people use commonly likes like it learn open c. d. for example
so on the top of this infrastructure what we do is we develop frameworks which are very task specific okay
so the idea is that combine all these numbers components you know if necessary
coming down to the version number to provide very high level features okay
examples of this framework to have a framework for barometric person recognition which
has plug ins for various modalities okay allows you to ramble metrics
on a lot of different um modalities so face and speak about metrics vein by my tricks as well
we always had a more we have another framework for biometric presentation attacked action
others when to talk a little bit about it yeah short wow
and it's a a framework as well for remote to forty plus years more murphy okay you want me to talk about it
so what is the task of this framework so the task of those remarks based the fine
they can protocol formats which are important for that particular problem for that particular task it okay
and something fine for the more blocks or the basic boss to the fine times
for such building block so this if you are working this program in this is
the same as kind of trying to the right place classes my problem okay
and to define the full and that is to see just looking mary speakers basically
and so that i spots which i'm pork in the context of that people tasks okay
of course it handles the idea was well between the blocks okay it's and provided looking interface so that implemented
is of critical function and employ a specific way to solve that particular task in just walking in
um little blocks off of code and then modified experiment the way they wish
here's an example uh we have the barlow framework okay so in
this case the ouster of the framework decided to do something
like that to change in this particular set that soaking then
device based classes for each of these tasks and implements
plug ins that allow you to actually plug in the database preprocessing projects
act together to actually create an experiment with experimental to change
was the framework is in place he is how we actually do work okay we start from
the existing framework and force try to reproduce the baseline which available within that framework okay
then what we do is that we forget project we create a new project that extends bills baselines
improvise new way to solve that task okay we compare because it's
so we easy compared the new results was the existing baselines
if you're not ready for publishing then use their iterate again starting from two okay you
think that you have a goal and you can actually probably if your paper
you promote a new project into the stable container of or softer infrastructure okay make sure you have
documentation make sure you had best units and we also incorporate that on a nightly chat
so make sure that it's constantly check to make sure everything is okay and the results that were published and still be reproduced
submit the paper and if the papers accepted basic with why the packages that we have from that people work
on the public index and of course on 'em and easy to use
on environment allows you to basically download the whole softer stacking wood shop i'm gonna show that late
that's of course very automated so we don't have to do much the only thing you have to do
is to tag it on to that and the rest is done by the machinery for it
well see if the papers not reading any well back to start to okay and if everything is done then you
start an expert you're done it's okay let me see that show you that in action so the show you
how much time it actually takes to try to run one over resigns is what we
explain to newcomers
okay so you have a common line okay what i'm going to run here for use just this little demonstration and basically
what we do is two steps basically follow the basic installation
of the softest i for for bob which is documented
and then with all the instructions on the document it all by your
face framework so this we're going to do it face modality here
face recognition system using eigen faces and a very small database of experiment runs i'm gonna run the full experiment now
there's no softer installed basically call is going to take care of all that
it's going to stall the whole software stack okay
that may include for example you know important levers by beep see if you actually knew that
label the basic stack is installed notes basically stole in the framework i actually want to work on
and is running the baseline
there i have my baseline okay and this is what we mean by making it easy and accessible for everyone to actually run things
huh huh
okay some conclusions for the sparks so we live in the world of experiment engineering okay
so if you don't get data and soft instructions means often very often hardly to reproduce material
of course represented can improve your impact and is it up the researcher doing
impact is it's very difficult okay you need a lot of infrastructure and of
course in a lot of work to make things work okay but um
if you you you you get it was an inside your your
community that everything you're doing a circuit usable then there's um
of course a lot of stuff to to to have the right but it also gets you an opportunity
to you know centralise work and avoid commanders in research okay so that is basically that you
instead of doing you know i'm going to the next big reproduced with that
i'm going to be recognisable person or crucible group and you put in
place all these infrastructure so that everybody can benefit from it okay was bought what we try to do is to come up with you know
uh huh that's so you know stuff um
solutions for all the problems that are surrounding the suspect okay and hope that you can try
one of our frameworks and and please if you do so give us feedback okay
they're very open to that noise to look at the problem you know and all which i just showed
what can be improved on it there's a lot of respect that can actually um people you
so first of all is that sometimes downloading and storing data for
running experiments can be apart as well so he not um
when you produce data somewhere you're not always also rise to distribute this data even outside your company okay
of course we're talking about more more about big data so the samples of the data sense we're using an increasing
so you need to figure out you know where even to store this you know running number of samples okay
and there is of course the problem of the stability and sometimes you're dealing with very
sensitive data for example forensic data can't redistribute that kind of data i think
if you're talking like big experiments like so and they don't want to distribute the data unless
you're part of that expert because it's tough to to actually handle the data distribution sometime
no software management can also be a problem okay installation can be difficult you can mitigated with tools like on the lesser just
showed you but still very high but you have to maintain softer stack to make sure using works it's it's quite tough
okay so we need oh always to have some by your if you remember structures
think about you know part of software and integration of things like that
and of course have to plan for years so this this means you need to
put in place and sort of redistribution mechanism just now also easy to okay
and of course as we talk more more about big data um one thing that people
don't talk about very often is that for big data you need be computing okay
sometimes even if i get my data and i give you my software you can't really have
the necessary computing environments or the the the the horsepower to actually process and chant
that that they'd okay so it's in a certain sense kind of use
so there can speech okay so what it is basically is a platform for repairs were research
in which basically you want in your web browser to do everything i just showed you
okay so the idea is that it levers on our experience on this idea coding publishing work full
okay and the question we asked ourselves well the building it was
basic how can we generalised is frameworks that we have
so we could deploy to any experiment to paul in the the main okay so this is how we came up with big
it integrates data and computing small web interface okay anything also
integrate supporting right there on the web together with your
experiments okay so can actually link between experimental results in
the experimental cool okay and the data as well
um we also devised a lot mechanisms to you know make sure that
you can keep your research private while you're working on it
but then to make it also easy that that it is reviewed one actually submitted for option
we had other aspects that we wanted to keep in mind while we're developing first of all i wanted to to make
it very very accessible that's why you only need a web browser to run it to make it into week leave
social of course i mean we're using a lot of social networks now
these why not make research social ritual is a good idea
sounds like a good idea but active so the idea is that you don't have to actually go
to millions of different papers to figure out what is the best result i can get
on this particular data set it can just type in search for that okay
um of course is taught about data privacy you know the idea that you
don't have to handle large scale databases and of course it ends
you because you're basically running on data which you can't really download from
our platform we can handle all kinds of different proprietary databases okay
of course wanted have assurance we basically wanted to make sure that at any point in time we could go
back to sit in experimenting action reproduce it okay and of course it to be completely open source
okay let's see it in action because i think that's the best way to
explain it okay so he and those were also i have i have logged in
as the user tutorial just the user we currently use for for controllers like
this and when you log in uh basically have an overview of everything that is happening to you and that have access to okay so you have
latest advances somebody shares an hour it's more an experiment little which you can have it on the last year
you can have you can set up your only the boards and say i want to try
this particular type of experiment with this kind of data base in this kind of algorithms
and keep them here and you can even be notified for that by email when the the ranking on this on the new the bouncy
okay so you know if somebody came up with a better solution for the problem you are working on and here we have a an overview off
every components that you have actually inserted into the into the system
now how is it is to actually run experiment here well it's just a few clicks away
you basically only experiments and the system going to load the list of experiments you had access to the because
it means your own experiments or the experiments that have been shared with you either privately or publicly
and then you can just type in a search for the experiment you actually interested also in this case
here i'm just going to reduce the same baseline ahead before so this is the true that's right
which i can face is only eighteen t. so i'm gonna select one and then let's say this one here
okay and is going to open me showing the results of that particular experiment i can see here what this else or the
other that expand didn't it was important on this experiment so there's a few scale or is there some plots here about
score distributions i guess there's hours a lot as well very well
how is it is to me to reproduce then i just have to click this button here for 'cause actor for his experiments
and basically there's going to copy over all the information is required to re run that experiment on my private area
okay but below that they can that's it and i can see in broad experiment i can see for example the
two chain that was used for actually running that experiment you can see the same paradigm that we saw
the park was bought so in the left to have the database for presentation the
to change and to the right here you can actually see the analysis box
i managed to get here we
okay and i can see some global parameters that the design of this experiment to the side or to run
also can hear c. and information about the softer stack that is actually used to buy that experiment
what kind of computing power it's actually going to use okay and this the person that
the there's also made one of the parameters of experiment available here so i can
just go and try to to to to change it okay and configure the experimenter
actually use our reasons within the platform and databases in a certain set up
so i asked myself okay the outer has made it work with seven components
but can nine
can i make the system work was let's say seventy five point so i just have to go there i put up
go down and clickable button
i have to change and experimental work
i can't really use those models
asking you know big ah
well it tells me to figure out the news into this name looks good okay that's cool
now the experiment is submitted to the computing far and it's it goes slowly running as soon
as computer resources are available for me there goes the first block is not trying to
to run okay once that is done the next lot unable so this all control but but from automatically okay
and as long as it's done the last block is going to come out green as well
so wait on the line like
five minutes okay so it's it's just read okay so basically what's happening here is that it's executing each of this
oh but it seems that i i i sat for those blocks here any one in the video um
computer okay one interview process and of course i can say that the whole experiments going
to use a softer stack but i can also the fine specific softer stacks
i kinda find specific softer stacks uh for every block i have so potentially can
run you know let's say one of the box in using one softer stack
another one using another soft just a completely different things there was most important for
that softer stuff i can combine different stuff together in the same expression
even different softer stacks which is something that would normally don't we've you're working on the desktop
is a nice feature
so now the scoring it's it's run already okay the experiment is done i can reloads
i can reload the page and then i can see the results for this particular experiment which i ran
okay i can see the new result remind us myself but wait how does that compare to previous
one well i can just say okay but for now show me everything that looks the same
as this experiment here and of course because it was handles all the details on that particular experiments
i can just browse down and see exactly how my new experiment compared to the previous
one which i had to run before so seven components i can of course short
so that they will hear and see okay and not doing as well as the previous one i can actually try to visualise
i can try to visualise the combined plots together okay and a fine i'm happy with that also
when is i think it is my work i can actually go and create complete reports
i can have complete unsophisticated reports okay the but for makes available for you
so i can write anyone of this so what is the report so basically this is the the links of buttons
going to generate you are permanent link and unique link that you can actually call in case into your paper
and the peer reviewer can come here and then browse down to the to the
to the report read some rich documentation maybe some references and stuff like that
that explains well a bit what they are saying okay can we name expense to make
them nice make sure that the plots are configured the way actually ones okay
from here the reviewer or yourself you can export as figure and put it into your lap tech
into your lapping document a word document or whatever you have okay and it's um
yeah so the reader can come back and say okay let me look at what is that experiment actually made off
okay so is is also can browse the execution details of that okay
oh this is interesting what this person use i can go there
and actually browse back to the actual our it's okay if necessary for look you know in search for whatever i'm looking for
okay i guess that's basically ends it i can talk more but yeah

Share this talk: 


Conference program

Welcome
Sébastien Marcel, Senior Researcher, IDIAP, Director of the Swiss Center for Biometrics Research and Testing
24 March 2017 · 9:17 a.m.
Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:20 a.m.
Q&A: Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:54 a.m.
VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 11:35 a.m.
Q&A - VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 12:07 p.m.

Recommended talks

Décision assistée par ordinateur: de la donnée à l’information
Dr. Dominik Aronsky , Université Vanderbilt (USA)
7 June 2013 · 11:58 a.m.