Embed code
ah okay so thanks a full invitation to talk in this workshop
um the organisers asked me to talk about what would you see political science at
e. p. f. l. in particular so maybe plenty of very interesting technical talks
uh today this is not one of them it's a it's a it would be just a talk about policy
but i think policy is very important in these matters and i try to use example
um uh of far institution not that there's anything special but maybe to give you a
uh the point of view of a a manager because in the that's what i am for the
time being a on these questions and and and his point of view is is interesting
okay so that the several parts the first part is do we have a problem okay and the uh
uh uh i don't have to convince you that we have one but it's interesting to
to sense the magnitude of what we're talking about when we say that we need to open science and open research
um the first thing is that by any may take if you look at it just from all numbers science is doing well
okay so we're we're doing more of it um people talk more more about it the number
of papers being published uh is increasing the number of research that is being funded
is increasing but that's the surface of things it's interesting to see actually what happens
behind this process so this is just an example i'm thinking from a medical studies in particular not not that
there is anything special with that i could have put out the same statistics from any of the field
but you can see that the pace at which we producing results seem to be increasing and
that's not a problem per se more science is good for everybody the problem is that
as you see sometimes the metric by which we evaluate the studies and reasons why we
producing so many studies are sometimes fraught and one of the problem is when
what we could call hype uh becomes an academic metric so let me just single out what i mean by that
here is a maybe you've seen this diagram before this is a
make us study on full lips that give or prevent cancer
okay so if you if you open nature or other than set every so often you see that very frequently
there are studies on what what kind of food you should eat if you don't wanna have cancer
okay and each red dot in this diagram is one such paper
no one such studies so we have on the y. axis you have a fruit
and the left x. is a um gives you that show also although the probabilities that this would uh
a half a five arises or prevents cancer okay so each thought is a paper
right and that particular figure the whole figure was taken from a tested it
so it's a study that takes all studies and lets them together
okay and if you look at it's one thing to one conclusion
that that that strikes out is that pretty much any fruit
both caves and prevents can so with the exception i think
of maybe a bacon who which probably always gets cancer
in pretty much any study and the only example made those i don't know which seem to be good for yourself
so at the same time the message that this is sending out is that we're screwed and was saved at the same time
okay and this is all published material by the way any of these for that is that is a paper in a very
uh you know this you use german but you can see that the conclusions
that you have to draw from these studies is basically on them
okay ah i'm afraid i don't have the video because we have only two
technical problem but uh i strongly advise that you look at this i don't know i don't know if you know this show
on h. b. also don't uh last week tonight with john oliver is a very funny show in a lot of people are watching it
they recently either whole twenty minutes on a scientific studies
and his point was that percentage didn't wanna i just met before is that
if you look at studies you basically find anything right so uh everything
and it's complete opposites any people that are in a very funny way so
going you shouldn't look at that but the message behind that is
is very painful in fact because if you think of it coming
in particular from that country at that moment in time
we are facing the situation with the scientific discourse can no longer be taken for granted
and the message that is being sent out by the simple examples that you know every
football cues and prevents cancer is basically an e. getting the notion of proof
right you and i receive an indication what we're being told that a fact
desire true force all we don't know it cannot be true enforce right
and now we're just adding more to this me that they are facts and identity facts
and in fact it's our job to just prove we oppose it okay but
in our daily business because of reasons which will try to see
it seems that we've gone very far from the initial objective so go
and watch these twenty minutes it's both funny and also enlightening
okay so now what is the problem coming in so more most of these that's fine
so what what can we see using data by the way from these studies
one thing that you can see is that if you do text mining on most of the scientific studies you see
that the simons of flashy finding and only reporting are making a lot of noise in the sense that
uh this is a a bigger and bigger proportion of
very very positive and flashy words in papers
and so if you open up any people these days you find those words navajo unprecedented breakthrough
it's it's it's hardly knew told anymore right and it's not a negative
we'll go buy yourself try and publish a negative result these days and if you have that right
there's there's there's a few journals and recently i think that uh i was it it needs what's your
there was a a a session on negative results but you know that's the exception so
it's a fact i fink fact the language uh uh of
science and not just the language of p. r. the
language in the papers is becoming overwhelmingly positive and fables hype
okay and in itself that could be a problem
and so there goes the uh the bombshell which i think was one of the initiator of of
of this questioning about science and open sign so maybe you don't know about
this but uh it's an interesting people to between in twenty eleven
i plan basically the the review of his feet we just happen to be oncology but then the same thing was done you know because
so we took fifty free foundational papers so i'm not just talking random papers
he selected you handpicked fifty free of the most important papers and
see all being published in cell major that's it that's it
any charges we produce the results and since this is some clergy bizarre
results will people will just tell you a we now sort
of understand the mechanism of cancer and how we can change things
very significant messages and what badly found out is that
in forty seven out of these fifty three papers those results which is not reproduce it
so the conclusions that that i've been very positively put forth by these papers not
be verified we could be true but it could be false as well
and if you just do the math that's an eighteen nineteen percent uh uh racial of papers
has conclusion we just cannot shows
and since then they see lawmaker studies of been a performing other field so if
you do computer science and if you think your immune to that think again
in in pretty much any field the ratio is between sixty to ninety percent and it's not too difficult i think to
to apply the same tactics to to your field and just discover bias of that you probably fourteen days back
so that's something a little bit fishy something a little bit wrong with
the way science is being done and be not just being reported
the it's not just a question of pushing it to journalist who over emphasise
claims this is coming from papers right out of the p. also
so how did we get to situation where ninety percent
of people's we publish cannot be trusted
so as an institution that's that's a question that we're asking ourselves because
the visual of papers and we have to understand why this
is the case and what we can do so one one set of things i think the most clones on wrong policies
okay so this for automatics everywhere in academia uh you can not church a piece
of work just by the fact that it receives a lot of citations
you can not claim the quality of the paper just because it is published in nature
another set of statistic is this publish or perish mentality just since we judge ourselves by these numbers
we put ourselves in a situation where the only thing that matters is pushing the papers very quickly through
and if you think of it making your work such that others can pick it up and to produce it takes a lot of time
if your we only search all these days it would take or h. to actually do your
work in such a way that you can enjoy everybody can we could use it
and you can read a lot of um analysis all comments on the web which
basically advice not to do the produce a good research if your your research
the problem is that if you're not researcher just like me you just
don't care anymore because there's no there's no stakes right so
if we don't do something fairly fundamental and the problem is not going to solve itself
the second thing which i think we have to talk about is the preview system
right this is the central tenet by which we still turn the quality of what we
do but i claim that the peer review system these days is completely flaw
and i'll give you an example but the most striking things is whenever
you write a paper on sunday to journal how many papers
would get on the people would actually judge the quality of your paper if you send it would journal will be three
one of whom will probably not read the paper
then and and and the this the size of the comments you receive for any of your paper
is anywhere between five lines in one page and more often on the five lines side
okay if you go to very serious conference there would be
up to seven or eight reviewers but very short comes
on the other end this hundred so maybe thousands of people out there what you have access should have access
to your paper and comment on it but whatever they do in your paper is nowhere to be seen
the first thing is that there is no incentives for usable data or even reusable code
right now i mean uh i understand a lot of people in this room
um have to subtract it's all try to have this practical for usable data and code but
that's almost a personal choice on so that's something to be done there as well
and so this place for the fact that research should be more open but if
we just say it's better not be you know the problem we'll just stick
so what are the incentives and what can we do such that we
can get rid of these problems and and actually solve the issue
so that's part to the top part one is uh your relies there is a problem
you sort of understand why the problem is there are two is what can we
so there's a number of things and and this is the list of things that are more at the level of my institution
so what we are doing this we're acting on two levels one is the short term one is the long but the first
one which i think is very important is that it should not be just a question of choice of the individual researchers
there should be an institutional message that these practised says on not negotiable
okay so i deeply for work you we have a new president and the new president just said
uh implementing open signs as that on a daily basis in the institution is is piloting
so is just slowly changing institutions such that uh we could practically daily basis and that's not on the ship
one thing you have to do in order to a lowly structures trashy but
in this is ease of implementation tools that would be talks about
simple cool so efficient but simple to use tools to actually do the producing research
in this workshop so that's very important i think we have two major force to the bodies too
institutions actually to buy an anti chlorine these tools on their campuses the second thing is the words
so um take the example of e. p. f. l. we
have awards for the best speech teaches us for instance
so that's good but nothing in the the what the certain that the work is what what usable
or that the data can be accessed or that the code can be accessed so starting
from this year there will be an award for the most people use abilities
and little by little all these that produce even work have to have
a pad off explaining how all the data all the colds all
the techniques in the work can actually be reproduced by others otherwise
they would not qualify as a solid teases valid for no
the last thing guess what we're doing here which i think we should also do it
to a very institutional level is systematic workshops so people should hear this message
to see what the tools are that would be them was here for instance finishes the important at that university for instance
there is a whole you're twenty seventeen is the year of open and so fraud
all your entire university is running activities on open signs and we put usability
and so that's that's the strength of institutional message that is the longer to
so that these are couple of things we can do right away
and that we are doing right now there's the longer term which are things we can do and things that's which we need help
the first thing is not whenever someone is shooting for promotion in
an institution and that would be the case in ours
you should not just document how many papers that published so now we're asking
volume researchers to documents their contribution to open signs in the promotion deals
so they have to tell us how the implemented this ethics in the
way they did signs during their tenure targeted p. f. l.
and only if they show have you record of methods than it would be uh that would be promoted
it sounds trivial if you think of it but it's not because that puts a
lot of buttons on the researchers so we actually is a message that
somehow it would be easier to get promoted elsewhere because you would not have to do this extra work
but if we don't if this message nothing's gonna change
a second thing is and i think that's maybe the most important thing you can not just to promote people like me
okay so you have to act on the younger generation so those who
get promoted but also i think those who start their research period
so if you take uh if you if you if you go for p. h. d. t. the city p. affair
the one thing you have to do is of course but which papers but you have to follow lectures or
lessons and in credits and if you look at them these are lectures in all the technical very specific
starting in twenty seventeen there would be lectures on the fundamentals
of open signs mandatory for every p. h. d. student
so no one will stop each teaches at the p. f. l. without obese and background
on what it means to do good usable research and what tools are available
and i think that's key to actually having these practised is diffuse in the way the whole institution discussed
the last thing does not only depend on us but it's already
on on on the way it's a strict uh funding regulations
the forest or not if you're looking for money before nationals implementing more and more plastic
policies on open access and soon data management plan the you is doing the same
and the you at the moment we use be nice but i'm just telling you okay
your research should be open access and you should have a data management plan
but the way you have to show it is just about often your report well we have one
but believe me trust me on this one in the next few years it would be much more strict than than that
right so you would have to probably demonstrate that all your data is accessible
and if you take large organisations like the gates foundation of the what caldwell contrast the from
the law research they have this year that's the gates foundation last month in fact
could very strict rules on uh the fact that every thing that is funded by the money has
to go through open access and what controls implementing the same think down to open data
it's all very significant messages because if you ought to receive money from these foundations you have to abide by the policies
so things are moving uh this beach in fact if you pick up the economist
that's not something you do very often as a scientist to you if you pick up the economist the editorial isn't open signs
so i think this year in particular is is is key this is the year with thinks things will start to change
so open data management open access code oppose it always apply just just a few words
about colour repositories because this seems to be the most review part of it
and and in this community in particular you use data and you put you scott
and we like to think of us because i'm part of the same community as you know we are light years ahead of
the others because we use get you know our code is up there and then write a paper with the link inside
is there a good let me give you a few examples where things are not that easy so
even if you actually release code with your paper
a recent study showed that after five years
i think it was eighty percent of the gold was not just gonna go in the sense that someone
would just pick up the code even if it's still available now could not reverted or recombine it
and if you think of it yeah of course it's trigger and there's a lot of
dependencies into go that it's not easy to do something that is completely self contained
and so just the fact of putting called online is okay but that's the that's barely enough
and the the the time the time live the time span of these goodies the evaluation walls
there is a a big initiative informs that you may have or the school
the uh uh uh a sustainable software one to something where they're actually
uh huh what what you're doing is crazy but what it said in a yeah well
the other thing is that the others it basically stocking up the whole get up
okay and crew and and creating the all the whole dependency tree of everything that is on the top
and and just making sure that in ten years from now you can basically a single
out any piece of go out and have a way to you on it
okay why is it important that's the second example is that more and more of what the others do this computation
you could always claim you know why it's okay someone cannot one might you want networking five years no that's okay i'll just
pick up the paper and then there will be implemented in the other way i think of it this way no
a lot of what people do in materials science although discovering is computer based
right and a lot of it is inside puppy it away softwares
so if in ten years from now you cannot he won the new
medical experiment that led to the discovery of a particular material
all that led to the discovery of a particular way of folding
to pull thing and you cannot physically do the experiment anymore
right and there's many scenes that are being produced today based on a numerical drug design
so if you can not access just a cold that is the backbone
of these experiments thing you could not could use the molecule anymore
okay so you could imagine no you you could make astral fee scenarios for instance where twenty years from now
a a a simple drag that we designed the ten years ago when no we're not be able to just
put use it anymore because would not be able to be on the experiment that critically depend on call
another more trivial example from an institution uh you probably all know
the the software called so it works
right it's something that a lot people used to do three d. models uh every university i
guess or every instead of technology buys these licenses and and and you buy a
educational licenses because other people use it in class
that's interesting by the way so it's a proprietary software but ninety
percent of engineering students are trained using it so we train
student to be locked with appropriate or the software already interesting
but it becomes fairly a funny in the sense that
you're not supposed to do research with that software if you have an indication license but if you have a
little research project in your lap the boundary between research and teaching is that because now what happens
certainly your in a situation where things that you have produced on a proprietary format so it work
that you may not be able to access if you cannot pay the license these companies for
instance now they have a new business models that they run off to universities and startups
and say well we know that you use our software but we also know that you don't have
the license that's because they used to take action what and this in on was money
so what happens is that these small universities and companies what they do is well you know we just ditch started work
but if you just ditched only work you cannot access back to the pfizer put just with his puppet away
there's only a fraction of a that you can export and then you well we've legacy that
you cannot access anymore and you just shoot yourself from things you have done just with
okay bob free i think this one is also important i think we have to rethink jointly the paper
so ah that's all too and the p. d. v. that's all the to the tool for community
so for one let's just consider things which are fact
science is a collaborative endeavour and it's more more collaborative
and the reason is very simple right it's it's getting more and more difficult to
actually produce results so it it it's it's better to actually team up
so teams get extremes so this is the teams that discover that expose on then they
they they have a nice paper which i strongly advise read in physical review letters
but if you wanna read it you first have to go for the twenty five first pages which are just the names of
the people on the experiments right it's five thousand or five thousand two hundred people that's what i mean by corporate
so that it someplace extreme you may lawfully but just look at the the other
numbers so this is your your average numbers that you can see that um
on the left you half the growth of articles we've uh a single authors and uh many authors you can
see the the the number of articles with several causes going faster and on the right track g. c.
the average number of authors spoke paper and you see that we'll know over three
and so we are we are more and more collaborating on tape so the number of water should spur article is increasing
and the number of articles of many many older is also increasing which is a sign of things to come
but at the same time the formats so the way we all we're doing all this the way we interact on the paper
is is there a different it used to be fifty years ago but the or it's not it's not change a lot
point in case last year there was the discovery of the first have additional ways
right so on the left hand side you have a einstein's calculation and paper where the
solution of the understand the question leading to gravitational wave was this was was
published that's the paper on the left a bit on the right that's the paper
where uh do you like to experiment was reported what's the difference between both
hardly any fact right right it's pretty it's the other one is a p. d. f. that's it
that's the only thing nice change in exactly a hundred years right if you want to print
the p. d. f. any would just be back to the situation on the left
just printed on old bay on hold your wish paper anybody could pretend you understand
so only the medium exchange and there's a there's a point to be made that academic publishing
schools extremely low on innovation right so we bred ourselves as the most of the people the most
innovative one if you look at the way we actually put a shoddy search all the way
people publish all the search because well a little bit slides that it's extremely innovative it has
not changed in twenty years and whatever else changes due to the very strong pressure
but on the other hand he should take again the example of gravitational wave that's much more you can
do if you look at the like the website instead of just going to the physically do letters
a website that you'll discover that paper as a link to to be double box
and the typical box contained or the process almost role data of the
experiments so you can't really run the legal pipeline or by yourself
and so everything is in there it's just not there anymore it's a paper
it's the coat good isn't bad and the data is accessible via
the should be done with book and everything is a single package this is very far from the people we had fifteen years ago
and so that's the way we should go more more often in the future
the second thing with the paper is the way the paper is evaluated and there i think that the p. did you
system which is rightfully at the heart of what we do somebody else to double check that we're doing things right
i think this one is really really really float i'd let picked up this little cartoon
that says you know most scientists we got the new streamlined peer review process has quite an improvement
and this is really how you feel right you have to when you when you finally put the last line you only take fine you're about
to send your paper you do feel on that whole with people you know just waiting to have you with whatever argument they have
the the only difference with reality is that the there will be six people they're just maybe three
okay let me let you process
s. not other people that's either oh um i was just criticising the the publishers
on the way they actually uh allow us to uh send our results not
but the way we collectively and by other provisions evaluate our research is also very very bad
and then we just take an example besides you pointed reviewers those
three guys know what does this say about your paper
right absolutely no one whereas if you go and buy music online or book on amazon
no you see the opinion that everybody who bought the same book in the same the same uh the
same the same music right you can check if they have the same taste as you are
right to by just looking at all the reviews that they have done and you can read that and you can decide by the circus
you know this is a deal good broke because this and that and that's what it's about and i went to just three people
also if you go to amazon all these websites you you have to be recommended
some content rather than just being a weighting the the reins on you
there as well there's a there's good news for the things which are progressing
uh you may have you may know of of the puppy or
a website we should although that's a very interesting but it's a collective
believe you side so uh it's it's mostly for the um um
my life science and a little bit of the competitions and says that that those are
just being the probably there and then you can uh review the paper is
of course you have to be extremely careful because if you go there and if you just open comments to everybody
there should be a quality system you know what it means to covent there and if you actually
go to to appear you see that sometimes it just calls nonlinear all the other end
this website i think there's some less than four euros existence
it has already the brunt hundreds of fake claims in paper and in particular little of claims of forgery
and sometimes the the bait even if if if if it's not all wrong paper the because very healthy
uh if you want to see how she did it take this paper the heroes of crisper
so crisp or is that really the most important technology can live signs in the you know in
the tickets to come and there's a very big fight on who owns the crystal patent
it's genie think so full but sake it's green it's it's with billions and billions of dollars
so what happens is that people from one institute which kept the patent decided to publish review on crisp
right and since uh that guy a calendar is very very well don't go to be published it's yeah
but when dealing is review it just the facts towards the fact
that most of the work to be done in his institute
given the uh giving everybody out there the impression that in fact that then should be going
to the broad institute because they had on the research so says the broad institute
but the paper was published in nowhere else that sell right which
is one of the most uh impact full journals on or
but if you go and check the the peer review the open to reduce side on that paper you see
that's some of the cool discover a if you look at the comment on the on your down
a journey filled now is one of the cool discover of crisper and she clearly mentions well you know in the paper they say
that they came to my lavender speed is that they didn't so they just blatantly lied about what they say they expect
right if there's no such platform there's no way of saying that everything
is cute in the spectrum and that's a very important issue
and the good thing is if if if you look at what's going on now we
are collectively moving out of the system and archive has been there for years
the biology suddenly woke up to use ago they sent by log i've
you probably don't publish an by archive go check it it's already much better now okay
uh there's links to other web site like this one thousand will can also
put on a lot of data so things are slowly slowly moving
and i'll i'll i'll just finish here i think if we have to also so
this problem we have to get into these shoes it also start realising that
the paper itself and the way we review working the way we publish work
this can change right now in in our disciplines the most complication ones
it's just not such a fun such of of an effort right tools like a jupiter notebooks where you can link
text colour which data this is that it is just
a question off the institutions and us collectively sending out the right message pants
we just have to embrace these tools and use them in our everyday research

Share this talk: 

Conference program

Sébastien Marcel, Senior Researcher, IDIAP, Director of the Swiss Center for Biometrics Research and Testing
24 March 2017 · 9:17 a.m.
Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:20 a.m.
Q&A: Keynote - Reproducibility and Open Science @ EPFL
Pierre Vandergheynst, EPFL VP for Education
24 March 2017 · 9:54 a.m.
VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 11:35 a.m.
Q&A - VISCERAL and Evaluation-as-a-Service
Henning Müller, prof. HES-SO Valais-Wallis (unité e-health)
24 March 2017 · 12:07 p.m.