Player is loading...

Embed

Embed code

Transcriptions

Note: this content has been automatically generated.
00:00:00
i think we're much ah let me start by thanking the previous speaker thing we sure are a lot of what you just said
00:00:07
um
00:00:09
to our quest to make our research more useful so everything that just that is really what we feel so
00:00:17
alright so
00:00:20
but i'm trying to go back to the to the roots of the the the fusion of repairs were research so basically paper
00:00:27
but not just the paper also the code the data and instructions to apply this code of the data
00:00:34
and uh be able to redraw the results that you had on the paper okay very simple idea
00:00:41
well let's take a step back and why would you do this as as the
00:00:44
previous because sad i mean there's a lot of papers coming out and
00:00:48
one important thing to realise is that we're leaving more more in experimental wage
00:00:53
okay the time where everything was hard to prove the masses kind of
00:00:59
more more over and work to see is that there's a lot of large
00:01:02
data sets and people were pushing a lot of experimental evidence nowadays okay
00:01:09
um so the conclusion of that is that it becomes harder to reproduce
00:01:15
the exact conditions all the claims of a given paper okay
00:01:19
um if you don't have the actual code okay if it's not an exact paper about
00:01:25
a mess in the mathematical method or it's not conception is simple enough to reproduce
00:01:32
and to be frank i mean who wants to go actually on and months of work and try to reprise of people like
00:01:37
that it's it's really tough you you know in here you basically have the mass you have some figures but you
00:01:43
there's no space for the other for example describe mathematical tricks that actually
00:01:47
took to actually be able to to completely implement isn't software
00:01:51
okay so we do it very often to to boost the impact of our
00:01:56
research okay so this means to increase our possibility of of usability
00:02:01
um this was pushed back to two thousand twelve but i'm sure it's true it's true true okay
00:02:07
um basically if you look at the the current number off of repairs
00:02:11
were such papers it's still very very small so there's an advantage
00:02:14
now doing that is because i can stand out in front of this very a big mess of papers that we have okay
00:02:21
only ten percent off tape this transactions in in image processing
00:02:26
provide source code and we know from it that
00:02:29
basically just provide a source code is not making it necessary reproduced will as it was highlighted before
00:02:35
and a statistic your work is also more valuable if it is reproduced were
00:02:39
senator k. thirteen out of the fifty most i think they present
00:02:44
in in company which is very important um um journal in
00:02:48
computer science provided the source code okay and the average
00:02:51
number of citations for people to do provide source code in in tape is sevenfold of papers that don't
00:03:00
no
00:03:02
this also here which is in the in the footnote um the find
00:03:06
six different levels of rivers ability and of course what we're trying to do here is to make things reproduce bowl
00:03:13
and easy to reproduce for people okay so that should take a very small
00:03:17
amount of the time i think the time here is not very important
00:03:20
but it if it takes a day of work it's fine okay and the idea is basically that we try to figure out solutions that work
00:03:27
for free okay that you don't have to pay for so get out of the
00:03:30
loop where you have models that are viewed on proprietary software okay but
00:03:36
if you look at it's there is there are elements which actually missing from this okay
00:03:42
the question the first question and i think this was already pointed out the fall
00:03:46
is how much software you actually need to make an experiment reproduced okay
00:03:51
do you need the whole of matlab to be installed only on platform we need it
00:03:55
stops being the the mass library to be available they need to be observed version
00:03:59
if you're using high some you know you'd is versions of the the the levers that you depend on the date weekend
00:04:05
um let pack and so what we actually need to complete all asks what about the machine do you
00:04:10
also need to you know make sure that the machine is available for the present reproduce it okay
00:04:15
now this for example uh we're talking more more more more about keep you process important we needed you machine to be
00:04:22
okay another important and all i think we can uses on that definition is
00:04:26
how much time do actually need to keep it reproduced will for okay
00:04:32
is it one day so you publish the paper and next day you know the thing
00:04:35
is not reproduce by anymore i think that's not really the anvil right so
00:04:38
you're thinking here more about a few we use to want to keep your paper
00:04:42
visible for for at least a few years after after uh it's been published
00:04:48
now if you think back on how many papers actually be a month painting at a certain point in time
00:04:56
if your student and you publish one or two papers but here and you're
00:04:59
getting a few papers a few years of papers that means for five
00:05:03
which i think is quite doable for a single person but if you look to
00:05:06
oppose doc it's more like ten abruptly there's more like thirty or fifty
00:05:12
and if your little to multiple groups you're talking about hundreds of paper that actually meant painting here so few years okay
00:05:19
you've actually talk about very long term mountains numbers numbers would explode of course
00:05:25
know how to organise this okay you can leave each by on itself and that's basically what is most done yeah
00:05:32
you research to figure out independently every student every post
00:05:36
talk to whatever they they think it's reasonable okay
00:05:40
of course this makes the work more independent from each other okay but
00:05:44
the work becomes very difficult to carry on somebody to pine okay
00:05:49
and uh this includes of course to compare results even sometimes within your same group okay
00:05:55
other boats as others bus reaching all the parties okay only one
00:05:59
you don't know because i'm not holding yourself this question okay
00:06:03
and of course you're constantly reinventing do we we basically have no reviews across the across the
00:06:09
the set of people your your your working us okay and the second option which we think is is better is to
00:06:15
actually put in place a set of tools and procedures that allow workers ability more to take place naturally within the
00:06:23
within your products and collaborators okay so that is that's we move into another angle in where
00:06:29
work is actually done together okay the results up here on the same grounds okay
00:06:35
and everyone is basically testing the same code base okay so this means of course less butts
00:06:40
people looking more eyeballs looking at the core would okay and if a fixed
00:06:44
is required somewhere that everybody's going to profit from that fix okay
00:06:48
there's a lot of softer use that's comes you spontaneous and there's an advantage
00:06:52
is that people because you have so just from softer community around your
00:06:58
your your area of work that new colleagues can basically start really
00:07:02
immediately to produce very high quality results okay but you trust
00:07:07
now if you try to do that you're gonna be face was a lot of questions okay first the very simple
00:07:13
one is how can i make sure that people can still download my contribution where should i place it in
00:07:19
the the second on which is very important is any point of time ah you made us contribution does is to work the same
00:07:25
okay and if it doesn't what have i done to break it was i was it somebody else or some software the panel
00:07:33
how do i fix it and how he redistribute that takes okay is the problem on
00:07:38
my site for somebody to pack my stuff but they do actually tell somebody
00:07:41
that they should fix the softer okay and this maybe mask works of using matlab
00:07:45
looking it does happen we do have a case but when that happened okay
00:07:52
so what does it take okay if you talk about i'm hopeless instructions which is
00:07:56
one part of rubber disability appear gonna need a website to host it
00:08:00
talking about availability here of your they have to think about long term availability of your of your softer
00:08:05
you need of course to create casts a that is going to help you answer the question does it still
00:08:11
work the saying if you don't make cast then there's no way you can answer that's simple question okay
00:08:16
and of course you need revision control system that going to answer the question what have i done to break it
00:08:23
okay i didn't have to figure out every distribution mechanism for fixes and improvements all the time
00:08:28
okay now there's the other angle of it which is the data itself of course you also have to think about a place
00:08:34
to host it and of course to think about users protocols in gonna be testing are you your making experiments was
00:08:41
of course redistribution taken for fixes you also patch data line you pension protocols you
00:08:45
make mistakes on your data collection so you should be entitled to fixed
00:08:50
we're not covering this data part you think there is a lot of stuff that has already been done i think we're doing it quite
00:08:55
a good job but the up as well already we do have a big border where we put most of our data sets available
00:09:01
i'm gonna talk about the first five he okay and introduce you something that we we we know
00:09:06
we've worked for the last five years well we just call bob okay what is bob
00:09:12
so ball is in a word a framework for reversible research okay
00:09:18
in short it's an open source to get so everything is free of charge okay
00:09:23
for signal processing machine learning specifically okay that's putting solutions for almost
00:09:28
all of the of all the problems that i've just mentioned
00:09:33
obviously you're all for it it's written on the mix of licensee plus plus it has a very permissive licensing scheme okay
00:09:39
it's one thing that idea whatever get let instance that whole school there which is very nice because
00:09:44
you can get cool display you should trackers you know computers integration in the planet as well
00:09:49
the packages i hosted on a public price and um um
00:09:54
package index which is um very available and there's lots of the huge community around it but not things that
00:10:00
it is a lot of unit integration test that we actually do it every come eat
00:10:04
and every piece of software we have most of the pieces of soft we have
00:10:08
okay and the check as well every every every every day once a day
00:10:13
if everything actually works to be still okay it doesn't break anything else on that sometimes people make changes and
00:10:19
that break some of your looking and this may mean results on your paper that you just which
00:10:24
and we also provide a mechanism than to the point everything so the whole softer stack using on
00:10:31
okay an overview so basically in this area of research most problem can be surprise surprise this
00:10:38
falls so you have a database way making castle experimental computer science on that to change
00:10:44
that helps in solving your problem creating a model that allows you
00:10:48
to analyse okay so the three three basic construction box okay
00:10:53
the database provides some sort of an abstract problematic access to the raw
00:10:57
data so basically the basic io the lousy to read data from
00:11:00
the disk okay also transformed data so the features that you want to
00:11:03
temporarily store a while you're processing dating site your to change
00:11:07
and as well the use of protocols how to actually use that data to reproduce the experiment
00:11:12
i have um well it's been a bit bigger okay i'm the user protocol is the
00:11:19
is the is the the problematic in entity that actually trying to experimentation okay
00:11:26
the to change what is the technique to solve that particular task okay typically will contain some preprocessing feature extraction machine
00:11:32
learning and so on so forth and the allies is basically set of tools that allow you to summarise
00:11:36
into single speller although deportment result and great great figures okay
00:11:43
so bob is actually sat very large set of components okay
00:11:47
that i haven't uniform well as we try to make
00:11:50
it as uniform as possible sorry test set of interfaces the follow basic construction units okay so we have
00:11:56
elements that help us to create such a database interfaces elements
00:12:00
where y'all signal processing machine learning and metrics of course
00:12:04
okay our basic honed install also you gonna get a lot of packages that
00:12:09
people use commonly likes like it learn open c. d. for example
00:12:14
so on the top of this infrastructure what we do is we develop frameworks which are very task specific okay
00:12:20
so the idea is that combine all these numbers components you know if necessary
00:12:25
coming down to the version number to provide very high level features okay
00:12:30
examples of this framework to have a framework for barometric person recognition which
00:12:33
has plug ins for various modalities okay allows you to ramble metrics
00:12:37
on a lot of different um modalities so face and speak about metrics vein by my tricks as well
00:12:44
we always had a more we have another framework for biometric presentation attacked action
00:12:49
others when to talk a little bit about it yeah short wow
00:12:53
and it's a a framework as well for remote to forty plus years more murphy okay you want me to talk about it
00:13:01
so what is the task of this framework so the task of those remarks based the fine
00:13:05
they can protocol formats which are important for that particular problem for that particular task it okay
00:13:12
and something fine for the more blocks or the basic boss to the fine times
00:13:16
for such building block so this if you are working this program in this is
00:13:20
the same as kind of trying to the right place classes my problem okay
00:13:24
and to define the full and that is to see just looking mary speakers basically
00:13:29
and so that i spots which i'm pork in the context of that people tasks okay
00:13:34
of course it handles the idea was well between the blocks okay it's and provided looking interface so that implemented
00:13:40
is of critical function and employ a specific way to solve that particular task in just walking in
00:13:46
um little blocks off of code and then modified experiment the way they wish
00:13:53
here's an example uh we have the barlow framework okay so in
00:13:57
this case the ouster of the framework decided to do something
00:13:59
like that to change in this particular set that soaking then
00:14:04
device based classes for each of these tasks and implements
00:14:10
plug ins that allow you to actually plug in the database preprocessing projects
00:14:14
act together to actually create an experiment with experimental to change
00:14:20
was the framework is in place he is how we actually do work okay we start from
00:14:24
the existing framework and force try to reproduce the baseline which available within that framework okay
00:14:31
then what we do is that we forget project we create a new project that extends bills baselines
00:14:36
improvise new way to solve that task okay we compare because it's
00:14:40
so we easy compared the new results was the existing baselines
00:14:44
if you're not ready for publishing then use their iterate again starting from two okay you
00:14:48
think that you have a goal and you can actually probably if your paper
00:14:52
you promote a new project into the stable container of or softer infrastructure okay make sure you have
00:14:58
documentation make sure you had best units and we also incorporate that on a nightly chat
00:15:03
so make sure that it's constantly check to make sure everything is okay and the results that were published and still be reproduced
00:15:10
submit the paper and if the papers accepted basic with why the packages that we have from that people work
00:15:17
on the public index and of course on 'em and easy to use
00:15:22
on environment allows you to basically download the whole softer stacking wood shop i'm gonna show that late
00:15:27
that's of course very automated so we don't have to do much the only thing you have to do
00:15:31
is to tag it on to that and the rest is done by the machinery for it
00:15:36
well see if the papers not reading any well back to start to okay and if everything is done then you
00:15:43
start an expert you're done it's okay let me see that show you that in action so the show you
00:15:51
how much time it actually takes to try to run one over resigns is what we
00:15:58
explain to newcomers
00:16:09
okay so you have a common line okay what i'm going to run here for use just this little demonstration and basically
00:16:18
what we do is two steps basically follow the basic installation
00:16:21
of the softest i for for bob which is documented
00:16:25
and then with all the instructions on the document it all by your
00:16:28
face framework so this we're going to do it face modality here
00:16:32
face recognition system using eigen faces and a very small database of experiment runs i'm gonna run the full experiment now
00:16:42
there's no softer installed basically call is going to take care of all that
00:16:47
it's going to stall the whole software stack okay
00:16:52
that may include for example you know important levers by beep see if you actually knew that
00:17:00
label the basic stack is installed notes basically stole in the framework i actually want to work on
00:17:07
and is running the baseline
00:17:18
there i have my baseline okay and this is what we mean by making it easy and accessible for everyone to actually run things
00:17:28
yeah
00:17:31
huh huh
00:17:41
okay some conclusions for the sparks so we live in the world of experiment engineering okay
00:17:47
so if you don't get data and soft instructions means often very often hardly to reproduce material
00:17:53
of course represented can improve your impact and is it up the researcher doing
00:17:59
impact is it's very difficult okay you need a lot of infrastructure and of
00:18:03
course in a lot of work to make things work okay but um
00:18:10
if you you you you get it was an inside your your
00:18:13
community that everything you're doing a circuit usable then there's um
00:18:18
of course a lot of stuff to to to have the right but it also gets you an opportunity
00:18:22
to you know centralise work and avoid commanders in research okay so that is basically that you
00:18:28
instead of doing you know i'm going to the next big reproduced with that
00:18:31
i'm going to be recognisable person or crucible group and you put in
00:18:35
place all these infrastructure so that everybody can benefit from it okay was bought what we try to do is to come up with you know
00:18:42
uh huh that's so you know stuff um
00:18:45
solutions for all the problems that are surrounding the suspect okay and hope that you can try
00:18:50
one of our frameworks and and please if you do so give us feedback okay
00:18:55
they're very open to that noise to look at the problem you know and all which i just showed
00:19:00
what can be improved on it there's a lot of respect that can actually um people you
00:19:06
so first of all is that sometimes downloading and storing data for
00:19:10
running experiments can be apart as well so he not um
00:19:15
when you produce data somewhere you're not always also rise to distribute this data even outside your company okay
00:19:21
of course we're talking about more more about big data so the samples of the data sense we're using an increasing
00:19:27
so you need to figure out you know where even to store this you know running number of samples okay
00:19:32
and there is of course the problem of the stability and sometimes you're dealing with very
00:19:36
sensitive data for example forensic data can't redistribute that kind of data i think
00:19:40
if you're talking like big experiments like so and they don't want to distribute the data unless
00:19:44
you're part of that expert because it's tough to to actually handle the data distribution sometime
00:19:50
no software management can also be a problem okay installation can be difficult you can mitigated with tools like on the lesser just
00:19:55
showed you but still very high but you have to maintain softer stack to make sure using works it's it's quite tough
00:20:01
okay so we need oh always to have some by your if you remember structures
00:20:06
think about you know part of software and integration of things like that
00:20:10
and of course have to plan for years so this this means you need to
00:20:13
put in place and sort of redistribution mechanism just now also easy to okay
00:20:18
and of course as we talk more more about big data um one thing that people
00:20:22
don't talk about very often is that for big data you need be computing okay
00:20:26
sometimes even if i get my data and i give you my software you can't really have
00:20:31
the necessary computing environments or the the the the horsepower to actually process and chant
00:20:36
that that they'd okay so it's in a certain sense kind of use
00:20:42
so there can speech okay so what it is basically is a platform for repairs were research
00:20:51
in which basically you want in your web browser to do everything i just showed you
00:20:56
okay so the idea is that it levers on our experience on this idea coding publishing work full
00:21:02
okay and the question we asked ourselves well the building it was
00:21:06
basic how can we generalised is frameworks that we have
00:21:09
so we could deploy to any experiment to paul in the the main okay so this is how we came up with big
00:21:16
it integrates data and computing small web interface okay anything also
00:21:20
integrate supporting right there on the web together with your
00:21:23
experiments okay so can actually link between experimental results in
00:21:28
the experimental cool okay and the data as well
00:21:32
um we also devised a lot mechanisms to you know make sure that
00:21:36
you can keep your research private while you're working on it
00:21:40
but then to make it also easy that that it is reviewed one actually submitted for option
00:21:46
we had other aspects that we wanted to keep in mind while we're developing first of all i wanted to to make
00:21:50
it very very accessible that's why you only need a web browser to run it to make it into week leave
00:21:55
social of course i mean we're using a lot of social networks now
00:21:59
these why not make research social ritual is a good idea
00:22:02
sounds like a good idea but active so the idea is that you don't have to actually go
00:22:07
to millions of different papers to figure out what is the best result i can get
00:22:10
on this particular data set it can just type in search for that okay
00:22:14
um of course is taught about data privacy you know the idea that you
00:22:18
don't have to handle large scale databases and of course it ends
00:22:22
you because you're basically running on data which you can't really download from
00:22:25
our platform we can handle all kinds of different proprietary databases okay
00:22:32
of course wanted have assurance we basically wanted to make sure that at any point in time we could go
00:22:36
back to sit in experimenting action reproduce it okay and of course it to be completely open source
00:22:43
okay let's see it in action because i think that's the best way to
00:22:48
explain it okay so he and those were also i have i have logged in
00:22:54
as the user tutorial just the user we currently use for for controllers like
00:22:58
this and when you log in uh basically have an overview of everything that is happening to you and that have access to okay so you have
00:23:04
latest advances somebody shares an hour it's more an experiment little which you can have it on the last year
00:23:09
you can have you can set up your only the boards and say i want to try
00:23:13
this particular type of experiment with this kind of data base in this kind of algorithms
00:23:18
and keep them here and you can even be notified for that by email when the the ranking on this on the new the bouncy
00:23:25
okay so you know if somebody came up with a better solution for the problem you are working on and here we have a an overview off
00:23:32
every components that you have actually inserted into the into the system
00:23:37
now how is it is to actually run experiment here well it's just a few clicks away
00:23:43
you basically only experiments and the system going to load the list of experiments you had access to the because
00:23:49
it means your own experiments or the experiments that have been shared with you either privately or publicly
00:23:56
and then you can just type in a search for the experiment you actually interested also in this case
00:24:00
here i'm just going to reduce the same baseline ahead before so this is the true that's right
00:24:10
which i can face is only eighteen t. so i'm gonna select one and then let's say this one here
00:24:17
good
00:24:21
okay and is going to open me showing the results of that particular experiment i can see here what this else or the
00:24:28
other that expand didn't it was important on this experiment so there's a few scale or is there some plots here about
00:24:34
score distributions i guess there's hours a lot as well very well
00:24:39
how is it is to me to reproduce then i just have to click this button here for 'cause actor for his experiments
00:24:48
and basically there's going to copy over all the information is required to re run that experiment on my private area
00:24:55
okay but below that they can that's it and i can see in broad experiment i can see for example the
00:25:00
two chain that was used for actually running that experiment you can see the same paradigm that we saw
00:25:06
the park was bought so in the left to have the database for presentation the
00:25:09
to change and to the right here you can actually see the analysis box
00:25:16
i managed to get here we
00:25:20
okay and i can see some global parameters that the design of this experiment to the side or to run
00:25:26
also can hear c. and information about the softer stack that is actually used to buy that experiment
00:25:32
what kind of computing power it's actually going to use okay and this the person that
00:25:37
the there's also made one of the parameters of experiment available here so i can
00:25:41
just go and try to to to to change it okay and configure the experimenter
00:25:45
actually use our reasons within the platform and databases in a certain set up
00:25:50
so i asked myself okay the outer has made it work with seven components
00:25:56
but can nine
00:25:58
can i make the system work was let's say seventy five point so i just have to go there i put up
00:26:07
go down and clickable button
00:26:12
i have to change and experimental work
00:26:16
i can't really use those models
00:26:19
okay
00:26:21
asking you know big ah
00:26:27
well it tells me to figure out the news into this name looks good okay that's cool
00:26:33
now the experiment is submitted to the computing far and it's it goes slowly running as soon
00:26:40
as computer resources are available for me there goes the first block is not trying to
00:26:45
to run okay once that is done the next lot unable so this all control but but from automatically okay
00:26:52
and as long as it's done the last block is going to come out green as well
00:27:00
so wait on the line like
00:27:04
oh
00:27:07
five minutes okay so it's it's just read okay so basically what's happening here is that it's executing each of this
00:27:14
oh but it seems that i i i sat for those blocks here any one in the video um
00:27:20
computer okay one interview process and of course i can say that the whole experiments going
00:27:25
to use a softer stack but i can also the fine specific softer stacks
00:27:32
i kinda find specific softer stacks uh for every block i have so potentially can
00:27:37
run you know let's say one of the box in using one softer stack
00:27:41
another one using another soft just a completely different things there was most important for
00:27:45
that softer stuff i can combine different stuff together in the same expression
00:27:51
even different softer stacks which is something that would normally don't we've you're working on the desktop
00:27:56
is a nice feature
00:27:58
so now the scoring it's it's run already okay the experiment is done i can reloads
00:28:06
man
00:28:12
i can reload the page and then i can see the results for this particular experiment which i ran
00:28:16
okay i can see the new result remind us myself but wait how does that compare to previous
00:28:21
one well i can just say okay but for now show me everything that looks the same
00:28:28
as this experiment here and of course because it was handles all the details on that particular experiments
00:28:36
i can just browse down and see exactly how my new experiment compared to the previous
00:28:40
one which i had to run before so seven components i can of course short
00:28:45
so that they will hear and see okay and not doing as well as the previous one i can actually try to visualise
00:28:56
i can try to visualise the combined plots together okay and a fine i'm happy with that also
00:29:03
when is i think it is my work i can actually go and create complete reports
00:29:14
you
00:29:16
i can have complete unsophisticated reports okay the but for makes available for you
00:29:21
so i can write anyone of this so what is the report so basically this is the the links of buttons
00:29:25
going to generate you are permanent link and unique link that you can actually call in case into your paper
00:29:31
and the peer reviewer can come here and then browse down to the to the
00:29:35
to the report read some rich documentation maybe some references and stuff like that
00:29:40
that explains well a bit what they are saying okay can we name expense to make
00:29:45
them nice make sure that the plots are configured the way actually ones okay
00:29:50
from here the reviewer or yourself you can export as figure and put it into your lap tech
00:29:56
into your lapping document a word document or whatever you have okay and it's um
00:30:03
yeah so the reader can come back and say okay let me look at what is that experiment actually made off
00:30:09
okay so is is also can browse the execution details of that okay
00:30:14
oh this is interesting what this person use i can go there
00:30:17
and actually browse back to the actual our it's okay if necessary for look you know in search for whatever i'm looking for
00:30:27
okay i guess that's basically ends it i can talk more but yeah

Share this talk: 


Conference program

Welcome
Sébastien Marcel, Senior Researcher, IDIAP, Director of the Swiss Center for Biometrics Research and Testing
24 March 2017 · 9:17 a.m.
Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:20 a.m.
Q&A: Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:54 a.m.
Reproducible Research with Bob and the BEAT Platform
André Anjos
24 March 2017 · 10:03 a.m.
VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 11:35 a.m.
Q&A - VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 12:07 p.m.
Making experiments on remote heart-rate measurement reproducible
Guillaume Heusch
24 March 2017 · 12:30 p.m.

Recommended talks

Can we tame mobile research in the wild?
Daniel Gatica Perez, Idiap Research Institute
3 Sept. 2012 · 2:31 p.m.