Player is loading...
Deeper down the rabbit hole of data, analysis, and inference errors and suggestions for digging back out
David Allison, Indiana University School of Public Health, Bloomington, USA
Friday, Sept. 29, 2017 · 1 p.m. · 53m 26s
Embed
Transcriptions
Note: this content has been automatically generated.
00:00:02
ah here i'm grateful for the opportunity so um the title of this talk to mitigate
00:00:08
recall i call deeper down the rabbit hole of data analysis and inference hours
00:00:14
and suggestions for getting back out and pour this idea of deeper down the rattles
00:00:18
reference of course the hours of wonderland falling down the rabbit hole work
00:00:22
we're up is down and black eyes widened is neither nothing is what it appears to be but also the idea going deeper
00:00:28
i knew what any was gonna talk about a little bit and i'm gonna try to build on some of his remarks
00:00:36
this omar disclosures i will read them all to you here you can read them yourself
00:00:41
and if anybody wants my slides just email me here i'll be glad to share them with you
00:00:47
so i'm gonna make some uh points about errors that occurred in the literature today
00:00:52
and i want to point out that we're all human and we all make mistakes and that includes maybe and so i'm
00:00:59
not just here to make fun of other people's hours um i wanna say you know what i make mistakes too
00:01:05
and as then he said earlier there's no shame in acknowledging
00:01:10
ones are is in fact quite the contrary once
00:01:13
you've been made aware of in or the the only shames to not acknowledge it and fix it
00:01:18
a few years ago my students got the thing i published a paper on b. m. i.
00:01:22
mortality in it we use a particular equation that was meant to correct self reported way
00:01:30
and um we showed up that using that
00:01:34
correction didn't make much difference in the formula for the corrections from another paper
00:01:39
and shortly after papers published we got you know you never wanna receive which is from the authors of
00:01:45
the paper was equation we used and they said we
00:01:48
cannot reproduce your results something's wrong or no
00:01:53
so okay now we're we're using an age data and so one of the nice things about it is
00:01:59
the data were all public so that's how they could show that they could we produce a result
00:02:04
so we said okay show us your code will show you our code we exchange code
00:02:09
eventually we're both able to figure out the mistakes it turned out both
00:02:13
groups have made mistakes that some mistakes the original paper in
00:02:16
in the transcription the equation we had some mistakes in setting up its use in our model
00:02:22
and so we work together remote uh a joint so rather more comment on
00:02:27
this me posted together and i think this points out several things
00:02:31
one is the value of raw data sharing the fact that
00:02:34
the raw data were out there the loudest expeditiously detectors
00:02:38
the second is that it didn't have the personal that we recriminations is
00:02:41
about the data that was about the code honest people made mistakes
00:02:46
and the third was we work together expeditiously to to correctly within state
00:02:50
look at the really embarrassing to correct it it's not that big a deal
00:02:54
don't really it overall change result important change to correct the scientific record
00:02:59
and so i wanna point all that out including and especially that i like to make mistakes and you know there's no shame in that
00:03:07
so some of you also know that we put out the obesity in energetic
00:03:10
so offerings every friday some of you probably got a few hours ago
00:03:14
if you don't hard on it and would like to be that
00:03:19
you know just uh you can look parrot that lincoln signed up or you can send me an email and i'll get you signed up
00:03:24
at three and every week we send out all about a hundred or so links
00:03:28
two papers in the literature describing something about obesity and energetic soon methodology and someone
00:03:35
and in order to put out that obviously we're looking at many more than
00:03:39
a hundred papers every week to pick what to put in there
00:03:43
and it's not uncommon that as i'm doing this every week papers come
00:03:47
across my screen and i look at this and i say
00:03:51
really could that possibly be true and then i'd go sometimes when i have time
00:03:56
i dig into little paper paper little more and i sometimes find now
00:04:02
it couldn't be true it's not true and we started noticing
00:04:05
a lot of these uh a couple years ago and
00:04:09
what we then did is we compiled all that we start writing some letters to the editor we got some papers vexed
00:04:15
and we actually had such an interesting collection of history with this that nature contacted
00:04:22
us and said we like to do a paper on this so we did
00:04:25
and so this paper came out which describes our experience
00:04:28
with not just detecting the errors which is part
00:04:32
of the issue i'll talk a little more about that for the big challenges is fixing the errors
00:04:38
and so unfortunately not all academics and novel journal editors
00:04:42
and here to the principles then you laid out in that he appears to an a. g. c. n.
00:04:47
which is if you find a mistake and it's really mistake admit it and fix it
00:04:51
and we have a lot of battles with waters and added and others
00:04:55
who don't want wood knowledge that a mistake is a mistake
00:05:00
so we'll talk about a few other kinds of common mistakes we see nutrition so that you can keep your
00:05:05
eyes out for them and so that you can help help think through how can we make this better
00:05:10
a denny's or we talk a lot about measurement so not to spend a great deal of time on and just to show you two slides
00:05:18
and here's the first one you've probably seen these maps of obesity levels by state odd nausea
00:05:23
so this is where i was from until very recently this is where i'm
00:05:27
from now um both places are places where there's lots of obesity
00:05:32
and people lot these maps they love rankings you know people like rankings in general and a lot these particular maps
00:05:39
and you have to always ask a good question sciences where the numbers come from where the data come from
00:05:45
and the way these where this confirms that the c. d. c. calls up a little more
00:05:49
than a hundred thousand people every year on the telephone and says to them among
00:05:54
other things but you're hiding much white and as you might not be surprised to learn
00:05:58
not everybody perfectly accurately reports their height and weight as nancy brown told us yesterday
00:06:04
so when you do that in order to do the state rankings you have to be making
00:06:09
the assumption that the differential reporting without the recording bias is constant across all states
00:06:16
and of course it's not exactly constant causal state so one of my body short
00:06:21
howard has actually measured haydn we data in many of these same state
00:06:26
and these are from the same point in time measured
00:06:29
writing wage on national represent what state representative data
00:06:33
and so you can see these different states and i just saw one
00:06:37
massachusetts down you think that's what you just talk last um n.
00:06:42
this is the ranking from the the self report data in this is the ranking from the
00:06:46
observe data and this breaking stays about the same which means that people in massachusetts are about
00:06:53
average honest compared to the rest of united states there no more or less honest then the rest of us
00:06:58
right you know now here is where sell them and it's a right down here and it goes way now
00:07:04
so it's just that at least with respect to hide away we don't about other things but with respect highway
00:07:09
people know about uh probably a little bit more honest than people in the rest of the country
00:07:13
and then if you look at others that go way up you know a year is missouri
00:07:17
you know it's quite the contrary probably a lot less honest then people the rest nice states
00:07:22
now you might say you know what who really cares about state make use of high to
00:07:27
make does it matter and i'm not sure does i'm not sure we should care
00:07:31
but if we spend as much attention as we do on it if it's important not to publish and talk about mouse will get right
00:07:38
and it's interesting i was talking to somebody on where your comments was talking to me about the hague suppose on
00:07:43
so we've got you know some scientists around the world are using these high tech machinery to figure out whether
00:07:49
they expose on exists and here is in two thousand seventeen we can't figure out a way people
00:07:55
so it seems to me that we we ought to really hold our
00:07:57
science too liberal higher level that we wanna be taken seriously scientists
00:08:05
alright maybe a little more precision though now the other thing is about support denny did a great job with this
00:08:11
one not gonna be labour i just wanna are point that we have a paper on this our own
00:08:16
comments of the mainly about almost exclusive about energy measurement not
00:08:21
i'll mention results work of food intake by sort
00:08:23
of work but i think here's this three point don't wanna make because we hear this a lot
00:08:28
and the key point is that just because the measurement method one has a hand is the best available does it make it adequate
00:08:35
so we hear a lot people say what david you haven't proposed anything that
00:08:39
what else am i gonna do i cannot use doubly label water in
00:08:43
twenty thousand people action yeah maybe that could be true you might not
00:08:47
be able to and they say it's all important that i study
00:08:52
this aspect of energy bounce in a large apathy melodic study it's really interesting questions yeah it is
00:08:59
and they say so the best thing i have is this that that you says invalid yeah that's true
00:09:04
and they say so well should i do well then uses that maybe just not ready to study
00:09:09
right if i told you i the best available space shuttle to take you to morris
00:09:14
and i could convince really was the best and i say but by the way it blows up ninety five percent of the time
00:09:21
do you wanna get on that special maybe so maybe i'm just not ready for the trip to mars yeah
00:09:26
maybe that's important scientific questions a lot we can learn by landing on mars but maybe i'm not ready to go yet
00:09:34
alright errors of design i think is another problem again danny talk about the observation start
00:09:39
i implore the thing about observation all at the genealogy is not that we should do it
00:09:44
but it's sometimes the over use or over interpretation of it and part of that is not knowing when to stop
00:09:52
so this is the quotation from daniel common nobel prize winning economists from his book thinking fast and slow
00:09:58
any says reliable way to make people believe in fall so does frequent repetition because familiarity
00:10:03
is not easily distinguished from truth authoritarian institutions of marketers have always known this
00:10:10
you do not to repeat the entire state of the fact we're idea to make it pure true
00:10:16
people will read who repeatedly exposed for example to the phrase the body temperature the check in
00:10:23
well more likely to accept this through the statement that the body temperature chicken is one hundred forty four degrees
00:10:30
by the way i look this up on the u. s. d. a. website much of coke at the body temperature which it is not one hundred forty four degrees
00:10:37
ah but the more you say at the more people tend to accept that is true
00:10:42
think about some of the many things you've heard over and over in time up until about a year or so
00:10:47
ago when we were all the many talks about got microbe iota the typical way to introduce such a talk
00:10:53
was to say there are trillions of cells of microbes in our
00:10:58
body they outnumber us by one to two orders of magnitude
00:11:03
there are approximately ten percent of our body mass is microbes and then yeah so people said
00:11:09
it all over again they didn't decide a reference anymore where did this come from
00:11:13
finally somebody go back in the nineteen seventies paper on the diary a sample one sick man
00:11:19
you know and and so you guys think we we hear these overall we to the point we forget to ask
00:11:26
so here's an interesting thing in the early nineteen nineties somebody did a study first one we
00:11:32
not in which they correlated in epidemiological observation study
00:11:37
the association between breakfast skipping versus eating
00:11:42
and obesity and what they found was that if you skip breakfast you were more likely to be
00:11:49
overweight or obese then if you did it relative risk about one point seven five or so
00:11:55
that's fine good study any might say at that point we need to replicate this let's
00:12:00
see the holds up very good idea so people came along the replicated it
00:12:05
and what and we round up all at the time a post doc with me did was he took all these ended
00:12:11
and accumulating that an else's mother would isn't that allies in the data as well what's occurring in real time
00:12:17
and saying how with the point estimate changing and how are competent changing
00:12:22
what you can see over time or more data start looking like
00:12:26
a point estimate starts changing parameters or we're still hovers around one point seven five
00:12:31
the confidence intervals gets smaller and smaller with significant here at this point
00:12:36
the p. about you use a lot of negative log p. value
00:12:40
it's already at this point at about ten to the minus three and so what you think about
00:12:48
here is that um we already knew conclusively that it was an association nearly nineteen nineties
00:12:55
at that point it was no more need to replicate vision epidemiological research the answer was there
00:13:00
we get right skipping breakfast is associated basically got it stopped no more study is needed
00:13:06
how often do we see at the end of a paper we always see and in conclusion more research is needed
00:13:12
when there's some whatever any conclusion no more research is needed on this particular question
00:13:18
that's really no more research was needed up at the logic observation nature by the end of
00:13:24
the nineteen nineties on the question of whether breakfast and something was a source of obesity
00:13:29
what was needed was randomised control trials to see if it was causal and yeah what happen is we drove that
00:13:36
and uh all down until the p. value when we start counting was ten to the minus forty two
00:13:41
right holding the inverse of adding cans number for the number protons in
00:13:45
the universe this is a little more precision then we need it
00:13:48
and it was not a good use of of our time it was not a
00:13:51
good use of our resources and more importantly wasn't getting the question answered
00:13:56
but what did it do by the mere exposure fact that continent talked about
00:14:01
we devoted our pages are time or resources to research that increase belief
00:14:07
without increasing knowledge and so to the point where people came
00:14:11
to believe that wasn't on the politically demonstrated fact
00:14:16
that eating versus skipping breakfast like the better way control
00:14:19
surgeon general had on her website until recently
00:14:24
alright now what the observation research we heard some good talks on that yesterday
00:14:30
uh one point one speaker said we've controlled for all possible can founders
00:14:36
i don't know how you control for possible to downers except the ram station that's one that i know that controls for possible confound
00:14:42
but the dialogue we're gonna do we believe the observation that pretty
00:14:46
much every so often goes we recognise that there the limits
00:14:51
the author then says yes there measurement problems yes that's
00:14:55
possible confounding yes there are other potential issues but
00:14:59
let me tell you about how good my study was it was a very homogeneous
00:15:03
sample they were all from uh switzerland and all you know very homogeneous
00:15:09
and they were all nurses so they all have the same socioeconomic
00:15:12
status and i measure things very carefully and so on
00:15:15
and think oh okay maybe that should make me believe it so i thought what
00:15:19
we took this to the limit what if i did that the near perfect
00:15:23
observation let the noise sixty so i'm gonna get a lot of identical twins or maybe
00:15:29
a genetically identical so all this concern about kinetic energy is gonna be gone
00:15:35
and i'm gonna get away from denny's concerns about measurement error i'm not gonna ask them what the
00:15:40
i'm going up observe them twenty four hours a day and measure everything that goes into now
00:15:45
and i'm gonna get rid of the socioeconomic status concerns because somebody put them all in the same housing you know
00:15:51
adam points out that sci stripes these things pretty quick and found a good point
00:15:56
we put them on the same housing will give them all the saying opportunities for fruit and
00:16:00
someone and then we're gonna see what happens what in that situation could i'd hop on
00:16:05
that my observation all epidemiological study done better than you could ever do
00:16:11
would give me the same answer that are randomised experiment would give me and i realised i've actually done this study
00:16:17
i've just done it in mike's nineteen people so one of our mouse studies we have some my spare reading i love
00:16:23
human some ice they're fed a little less and some ice the randomly assigned to be fed a little less still
00:16:29
and within the ad would be improving actually also the other groups there's some variation
00:16:32
how much the mice shoes to eat so now i've got two elements
00:16:36
i've got an attribute grouper mice shoes to you more or less spontaneously and
00:16:42
i've got an element were i randomly assign my steve more or less
00:16:46
and so i can carly how much the mouse choose to you without comes and
00:16:50
then i can look at the experimental effect of assigning mice to different amounts
00:16:55
and here's the experiment or fact this is been shown many times if
00:16:59
you freedom of moore's ever always we let them go beside where
00:17:03
they don't live as long as if you colour to restrict them or make them lose some weight so no big surprise there okay
00:17:09
but here's the association of self selected energy intake in
00:17:14
these different groups with longevity or like spam
00:17:18
it's the exact opposite and statistically significant in both directions
00:17:23
so that if i were to do this as an observation of starting with
00:17:26
the best measure menu could ever achieve better measure the garbage even humans
00:17:31
i would conclude that eating more causes you
00:17:36
to live longer but if i'd say what is the effect of making animals
00:17:41
the last i say eating less causes you to you long live longer
00:17:46
and what i think the shows is that there is something fundamentally different about
00:17:51
being randomly assigned to things versus self selecting even when you tightened up your measurement and
00:17:57
got genetically homogeneous group rides all inbred mice the role uh in probability genetically identical
00:18:04
and so i think it really suggest that there is no perfect observation started i guarantee matt
00:18:09
how well you do it that you get the true estimate of the causal effect
00:18:14
some people have said was what this just probably means david is that
00:18:17
these animals that choose to eat more to hold your animals
00:18:21
the cigarette animals choose the lesson that to try and it's amazing yeah
00:18:26
that's the point that is a confounding factor which is not
00:18:30
experiment with the lights are also errors of analysis
00:18:37
one of these occurs in that analysis some other analyses are as in remote can once
00:18:41
said that analyses are easy to do unless you're doing it well that's hard
00:18:47
so that analyses have become this this quote unquote cheap way to get a publication the software is out there
00:18:55
anybody can go in and type in their effect sizes or the numbers and the software will dutifully
00:19:00
spit out stuff but the problem is first you have to know what the type then
00:19:04
and in particular you have to know which variances to include and that seems to confuse a lot of people
00:19:10
means most people get right variance is not so much
00:19:15
this is a paper that came out long time ago twenty years ago now
00:19:20
and we came across my desk um actually we're responding to a paper by these guys out when
00:19:25
it came across my desk i know something about the effect size in this uh thirty
00:19:33
this net analysis was reported as a whole wednesday standardise mean different one point nine six
00:19:40
and remembered something uh when this book labels and that any also signed distributions are standardise effect sizes rising
00:19:46
maternal slogan domains also standard research it is unusual for the magnitude to be as big as one
00:19:52
why where these big one point four and extraordinary would be as big as to
00:19:58
so now we say not only is one study got something almost as big as to what the average across an
00:20:04
entire body of literature of hypnosis for weight loss is as
00:20:08
big as to just really could that be true
00:20:12
so there weren't that many papers so i said the mouse fate with my post archetype cole porter papers out
00:20:18
so people the more on i said let's recalculated not that much work was it that doesn't papers all
00:20:24
we recalculated we found that there were errors in more than half the paper calculations
00:20:29
we fix the errors and the one point nine six in down to point two six
00:20:35
and if you actually throughout a study that was look quite questionable
00:20:40
um yeah actually came down to point to one wasn't even statistically significant anymore
00:20:45
so what we see is people um seem to make lots of mistakes and not an l. c. so
00:20:49
take note analysis with a grain of salt please don't ask you graduate student to do one unless he or she is a statistician
00:20:56
please have a professional statistician involving human announces just because your students as they can
00:21:00
turn the software on it three doesn't mean they know what they're doing
00:21:04
um any reading that analyses look at the effect size is a lot of times we pick these up just 'cause i
00:21:09
looked margo mm one of them's already over here what's that one doing over there we looking we find mistakes
00:21:15
that's exactly what we've done here one came unglued commanding recently took a man and helps with weight loss
00:21:22
and again i looked at it held up this little graph there about a half dozen effect sizes
00:21:26
one of them was way off on the right i said to my latest post doctor shawna
00:21:31
colds b. s. additional it'll pull the paper down let's look at it again she
00:21:34
came back she said i can we calculate five but not the six
00:21:38
so we contacted the original authors they said well we have some original data
00:21:44
from the first daughters who had the paper we met our wise
00:21:47
and a simple we send it to us they got permission to do with the centre was
00:21:50
really original data it was just calculations of means in estes looking at these calculations
00:21:56
and i couldn't figure out what was what so sent the to kong whatever starters the parking you figure this out
00:22:01
so let me go work out of what comes back and on our any says doctor asking me think i'm crazy
00:22:06
i see okay he says but take a look at this and he says you see other calculate the mean
00:22:11
difference here you know mean it three minus mean posters yeah it's just a look at the standard deviations
00:22:18
and he said if you ignore the signs it's the same as the
00:22:21
standard deviation three minus this innovation pose is it yeah oh
00:22:27
and he said but you can't even make it stand aviation i should know unless
00:22:30
we work with imaginary numbers here your nose like a square root right
00:22:34
and he said i think they just drop the sign which is with packed exactly what they did so they
00:22:39
didn't know how to calculate variances they thought that difference isn't stand deviations was the standard deviation of different
00:22:46
screwed up the results we help them fix it again it's very nice they admitted they made a mistake
00:22:50
we work together we published a little note together fixing it and what was significant is no longer
00:22:55
significant so good command in is not there was not significant evidence that it causes weight loss
00:23:03
this is one of my my favourite ones lately this is the darling design art
00:23:09
public health um people wanna do community into branches school based interventions and so on
00:23:15
if you're going to school for example when you say we're going
00:23:18
to um give a fruit and vegetable promotion program to kids
00:23:24
it's hard to go say are you get the fruit or vegetables and you don't need any future vegetables that's a real strange
00:23:30
so what usually wind up doing is you you take an entire school and you say we randomised this whole
00:23:35
school to let's say get fruits and vegetables instructions about fruits and vegetables or physical activity whatever it is
00:23:41
we go to the next school they get randomised to something difference all through randomised at
00:23:45
the level school work county or city or building for classroom or physicians cactus
00:23:53
and the mistake people often make is that the forget that they
00:23:57
randomised at the level is think will cost school the office
00:24:02
and then each week the individual subject individual person as the unit of analysis and
00:24:08
if you do that your statistics are as they say in yiddish for cocktail
00:24:13
um you can figure out what that means so um
00:24:18
i'm not gonna go through this but we published a paper there's the reference in ages yeah
00:24:22
and talking about cluster randomised trials and the special things that need to be done
00:24:28
and here's an interesting example so um this is a effects with twenty month
00:24:32
cluster randomised school based intervention trial on b. m. isolate boys and girls
00:24:37
and the authors wrote as only two percent of the variance and be on my waist circumference
00:24:43
was explained by group we did not adjust for clustering analysis
00:24:47
now to set you've gotta would just close to new houses you don't do it's not down they said well oh so small
00:24:53
the intro class correlation was only pointed to two percent it's so
00:24:57
don't need to worry about and the side a particular reference to support reference but i've we've got about second fido
00:25:05
what's interesting is if you go back to that weapons you actually read it it directly contradicts
00:25:11
them it says even small into class correlation coefficients as small as point o. two
00:25:16
can really screw up your results can we still simulation of this and in fact we show that
00:25:23
with their sample sizes and so on this is the we are estimated type one o. eight
00:25:28
if you ignore the intro class you can order clustering as they did with this level of intro class correlation
00:25:33
right so instead of having a point of all level they were really operating
00:25:37
at somewhere between the point one all and point one five level
00:25:41
and so many of the things that you readers cluster randomised trial results
00:25:45
or in fact just plain wrong many the things that are purported to be
00:25:49
significant or not this is a big problem again if your journal letter
00:25:53
i would ask you keep a look out for ice services so see that are for a. g. c. and for i. g. all for obesity
00:25:59
and on all three of those journals it so just like a standing or they see
00:26:03
the work last they just you send it to me were the other statistical editors
00:26:07
just to to take care of it otherwise we get crazy result about one for good time
00:26:12
right here's an example of what happens when you don't send it to the statistic letters first
00:26:17
so this was a cluster randomised trial of of gardening intervention that was
00:26:21
probably still be city that had to be retracted because of
00:26:24
the analyses were done correctly in the results didn't hold up
00:26:31
um cluster randomised trials and attractions come together to be particularly sticky
00:26:36
this is an interesting story up paper came out from
00:26:39
a brazilian group energy a journal called obesity fast
00:26:44
and this was um a methodology paper telling people about some
00:26:47
of them not the logic issues in cluster randomised trials
00:26:50
and what they should be concerned about how to do certain things and as i read it i thought gee i don't think this is right
00:26:57
and not only do i not think it's right at some simulations and some men analyses in a make a point
00:27:02
is it i don't think you need the simulations are mad analysis for that i think this is noble
00:27:07
from first principles mathematically this is just that should be obvious to any trends that station
00:27:13
and so i sent around so my buddies who are you know experts in cluster minimised process it
00:27:18
you think i've got this right this really yeah yeah you gotta write this paper that make any sense i said
00:27:23
okay so let's write a letter to the editor so we wrote a letter to the and we said
00:27:28
support the look a cluster randomised trials but respectfully the statements made
00:27:32
by the authors are in contradiction to mathematically provable facts
00:27:36
and that unknown in literature and we really think there for your to retract the paper because is putting out misinformation
00:27:44
and the latter took our paper sent it to a
00:27:48
another statistician independently boss who came back to us
00:27:52
these guys in alabama maybe crazy but they know what they're talking about
00:27:56
on this particular issue and um the paper should be attracted
00:28:00
and sent it to the authors and your preset no we don't want retract our paper
00:28:06
and the other said but you're wrong should be retracted and they should
00:28:10
know and the other was a forty track but chose not to
00:28:13
interestingly the other then wrote a whole editorial describing this a retraction watches little story about
00:28:19
the whole thing here so i think this points out to really interesting things
00:28:24
one is how confusing cluster randomised trials or to people that
00:28:27
even mythology still confused about them and the second is
00:28:31
the problems we have with the journals where they'll really wanna
00:28:34
take responsibility authors and editors seem unwilling to take responsibility
00:28:40
you know if you hang around in places like alabama you might hear about singer named eric church
00:28:44
who's gonna popular songs as i learned that from a three year old and refrain is
00:28:49
when you're wrong you should just say so i learned that from a three year old so i think we need to
00:28:54
get our editors in ourselves in a little bit harder just being will say you know well mainly staple wrong sorry
00:28:59
all the paper correct the paper it you know it's better to correct the record uh they're not
00:29:07
i and number one of the problems we have is what we call the dunes are doing
00:29:10
this is an acronym one of our our guys made up for difference in nominal significance
00:29:16
we see this a lot in randomised trials often
00:29:19
small randomised drastically exact dietary supplements nutrients
00:29:24
um and somebody doesn't get a statistically significant result
00:29:29
when they compare the two parallel randomised groups
00:29:34
or maybe they just don't know do that test and so instead what they
00:29:38
do is they first cast whether the treatment group change significantly from baseline
00:29:43
and then they cast whether the control group change significantly from baseline and if
00:29:48
the treatment group change significantly from baseline let's say please point four nine
00:29:53
and the control group didn't let's say p. was point o. five one
00:29:57
they say ah this group change this group didn't i
00:30:02
have an effect it's significant i've shown advocacy
00:30:05
and most of you immediately realise that that's not about task if you do the math of it you relies wonder
00:30:12
the worst case scenarios with equal groups um you have you can have a fifty percent type whatever rate
00:30:18
in that situation i would on equal groups you can get the type one error rate because two hundred percent
00:30:24
um martin planned and doggone wrote a paper in a. g. c. and
00:30:29
describing this if you wanna have a detailed discussion of this
00:30:32
your other papers and letters well and here's another paper that pointed out
00:30:37
it was these guys put up april flax seed they show that
00:30:40
flax seed seem to have beneficial effects on cardiovascular rest and again
00:30:45
as many c. of looking at it said gee this
00:30:48
doesn't seem right seems like they made the denser and then i know says
00:30:52
little windy said we have poster or data here publicly which i'd martin
00:30:57
for so we picked up the raw data we analysed it i'm looking at the correct between groups that it was not even a with
00:31:04
of an effect and so we wrote to the other and then here both the other in these people did
00:31:10
the right thing and they said you know what we're sorry we names they would use the wrong analysis
00:31:15
and the paper was retracted there's another one where this came up recently
00:31:21
this paper just came out pediatrics in there's a letter correction
00:31:24
percolating through pediatrics system now so this be corrected um
00:31:30
i first these data theme okay since the trial net mormon it'll
00:31:33
be sitting in in you blow up repute roberta pupil children
00:31:38
and what you see is easy these different weight losses in these different groups possible met one and keep you look you blue
00:31:45
and here's the difference in different so in the creek you will group
00:31:51
the net forming group last well point two more cagey that could
00:31:56
use without numbers um then uh then the placebo group
00:32:00
and the same thing occur in the pure will just okay that's fine consistency
00:32:06
reach last point two cages more and then the statement comes out
00:32:11
that formant increase the b. m. i. z. score improved inflammatory choreographed relate obesity parameters
00:32:17
in pretty people told the banana pupil talking hands the differential response according
00:32:22
to puberty might be related to the dose of my form
00:32:25
so i imagine you're talking to a clinician and use arrives import paper that just came out you have to
00:32:29
think about the differential response to my foreman if you're dosing or preview we'll versus people to others
00:32:36
oh thank you professor hobbies that differential sparked well me tell you if you know what i would do or zero
00:32:46
one the differential response that doesn't exist well because what they did is they basically said well we got a significant
00:32:52
result in one group and not the other and then drew this crazy conclusion and so that's been corrected now
00:33:00
one of the other things that i see in the epidemiological uh literature and sometimes in the i'm allergic to is relying on
00:33:06
statistical ad hoc reason i think it's important to uh be
00:33:10
on the look out for the distinction between assertion improve
00:33:14
and so some of the things he's seen the epidemiological traffic a lot eliminating really getting
00:33:19
the uh my mortality you know he's wishing papers on on if you're interested
00:33:23
eliminating biologically um implausible values in child waiting hours he's
00:33:29
and eliminating implausible food intake analyses multi supported intake so because
00:33:34
we were talk a lot about supporting take your thought
00:33:37
one might mention this one so comments thought is to say well i know that the
00:33:41
salt purported energy intake so not so good and people turned on the report
00:33:46
so i'll use some rules for the finding who's really implausible on the
00:33:50
reporter and is a rule culpable bird will that some people use
00:33:54
and one side to find this subset where their voices also
00:33:57
really implausible i'll throw them out of my analysis
00:34:01
now redo my analysis and the presumption is now my
00:34:04
new analysis will get results better better more closer
00:34:08
to the truth then the original analyses and so then i can be more confident than those results
00:34:14
and the question is is that true and it's interesting 'cause you say well i sort of have some
00:34:18
intuitive appeal to it but where's the proof of this is their model can someone right out
00:34:23
some parameters for me in some equations that show that parameter estimates get closer to their
00:34:29
true value when this is done them when it's not done and i haven't seen anything like that
00:34:34
so we just did a little analysis where we took some real data where we had
00:34:38
uh objective measurements of energy intake by doubly above water rip stop report measurements of energy intake
00:34:44
and then we had some um we're blood pressure measurements just to look for something uh correlated with
00:34:50
and here's the slope of the regression of blood pressure on energy intake when you
00:34:54
use energy intake by double double whatever object that's about point six nine
00:35:00
here it is when you stop work it's about one two three
00:35:03
to four when you eliminate subjects by this goal were rule
00:35:07
it actually gets worse not better now i'm not saying that's gonna happen in every case what happened in this case
00:35:13
and what it really shows is that we don't know what can happen unless someone really writes this now
00:35:18
so i think a real thing we need to be um on the look out for these
00:35:21
kind of statistical ad hoc agrees that people use particularly in observation epidemiology with the state
00:35:27
here's this thing on throwing out there ought to make my data better without approve that actually has particular statistical properties
00:35:37
and the general question for me that i think is uppermost but his little joke
00:35:41
under certain phones the statistical consulting department to form them on doing the study and i'd rather just do my
00:35:47
own statistics so i don't need your help i just wonder if you can suggest a good statistics text
00:35:53
and the consulting statistician says i'm so like to call the voice was gonna do brain surgery can you suggest a good text on that
00:36:01
and i think we really have to think about i won't ask for short hands on this
00:36:05
but you know i bet a lot of people in this room are trained in physiology
00:36:09
in nutrition in psychology can you see ali g. and a few other things like that
00:36:14
where we all got our one warm may be to statistics courses in graduate school
00:36:20
and then a few years later we were getting a rusty on the software
00:36:25
and our up and coming grad which do not post doc was getting good at it and here he knew had turn as p. s. s. on
00:36:32
and you know anybody could turn s. p. s. is on it some ice you pointing you click in it with all these tables and so we think you're
00:36:39
the kid knows what he or she is doing they say they know what they're doing so i get the data the kid
00:36:44
i said allies it's a cage do a good job i get professor okay good and that's the quality control checks
00:36:51
and i think i've done that right i i was a kid sometimes them the professor and
00:36:57
i think that may then maybe in it was a day when i was
00:37:00
okay and i still think that is the norm in many many environments
00:37:05
i don't think that should be the norm when we really care about the regular for dayton will drop policy on it and we're gonna
00:37:11
you know have advocacy based on events or by hours of recording so
00:37:20
here's you know something i pointed out yesterday the facts matt
00:37:23
um assertions are interesting but the facts matter
00:37:28
this is from a letter more coke and i wrote in response to a
00:37:32
paper that were this title in john back in two thousand ten
00:37:36
and a group are really cool paper actually about phantoms good paper still is a good paper
00:37:41
and in this paper which they quantified how often people use spin to find is kind of
00:37:45
distorting were exaggerating results a little bit yeah you didn't get
00:37:49
a statistically significant fact in your randomised control trial
00:37:53
they made a statement they said our results are consistent with the idea
00:37:58
something like this that um financial conflicts of interest bias results
00:38:04
will lead to more spam something ones i so that doesn't sound good i mean to look at these data
00:38:09
so when i i could not find anywhere in this table in the paper
00:38:15
the data supporting yeah so i email my from market to market and maybe i'm going
00:38:19
blind can you find this thing is i don't see it anywhere the pink
00:38:23
so we wrote a letter to a jam we said this is an interesting assertion we can't find the
00:38:28
data you know the authors say they collected data on whether this is industry sponsored or not
00:38:34
but we don't see in the paper could they i could they show was the result the office came back and said
00:38:40
thank you for this comment our statement was an overstatement there was no significant association between uh
00:38:48
funding source and you suspect so not only was the low significant association
00:38:56
between funding source you spent by the fact that the authors
00:38:59
had spun the authors had stated that there was incorrectly this is a paper that just came out a few months ago
00:39:07
and this with uh a paper looking at discrepancies between the
00:39:12
major endpoint in a randomised control trial that'll be city
00:39:15
versus what was registered so so for example somebody registers the trials as my main endpoint is body fat
00:39:22
and then the paper comes out and they say our primary endpoint is wasted conference see something like
00:39:28
that i think um maybe they didn't get a good result them by fatty switched away circumference
00:39:34
and it's okay where you've wasted conference but just tell people you switched you know just be honest be transparent is um
00:39:42
so we looked into this nice again this is interesting because the state was the industry funded studies
00:39:48
were the most likely to have a major discrepancy with this is that it's done some
00:39:52
different history and we go look at these data and we'll look at the table
00:39:56
i thought it looks to me like the number for industry lower than any other group
00:40:02
then again i showed some my friends i said am i going blind them i confused him upside down
00:40:08
have i gone for the rabbit hole and they said oh no not as far as we can
00:40:12
tell it looks like industry funded studies all the least likely studies have a discrepancy between
00:40:20
the registration and the publication so we wrote a header and the authors have come back and said
00:40:26
alice in college or correct our steak was exactly opposite to what each of them we apologise
00:40:32
so we got a really these things gotta look at the data don't take everybody's word for
00:40:36
it when they say it is um incontrovertible that funding sources associate with this or that
00:40:45
our headlines i want the labour it because i know we we all know the disconnect between the headline
00:40:51
in the study is often crazy it's just examples which might ask where is this come from
00:40:57
well some of it is an hour observation all studies this is an a. g. c. and little little
00:41:02
old i think maybe better now a. g. c. n. i. d. o. b. c. journal nutrition
00:41:06
this was the frequency of studies finding an association and reporting it as
00:41:11
causation and you can see in one journals higher than fifty percent
00:41:17
talk about spending his non industry funded studies from the c. d. c.
00:41:23
um effect of healthy schools problem uh problems of overweight obese in california
00:41:28
the press release dates now is the aligned celebrate a ten year anniversary in new period you study confirms
00:41:34
we are delivering on our mission of reducing the prevalence of childhood obesity
00:41:39
an important means of supporting schools are reducing obesity that sounds really good well people be testing is we're delivering
00:41:46
what the data for this years from start analyse showed no difference between
00:41:51
healthy school problems causing control schools in overweight and obesity problems
00:41:58
further down the rabbit hole
00:42:01
and yet they say how peaceful probably appeared report means of supporting schools are reducing obesity
00:42:07
there's no connection between the data and the conclusions coming out of the c. d. c.
00:42:14
here's another one the impacted area based initiatives on his collectively trends in deprived areas
00:42:21
i think the prime target district showed significantly part of changing walking trend
00:42:27
um and intervention the trend uh deprive audio physically larger compared to the
00:42:32
rest no i haven't but not compared to other deprive disputes
00:42:35
for cycling in sports neither deprived just it's not controlled just showed significant trend in change
00:42:40
for all the time for selectively outcomes trend changes when
00:42:43
not related need tentative environmental interventions think well that's
00:42:47
a lotta now is loud nothing going on here right this violates pennies principle that something must happen
00:42:54
it's okay sometimes you gotta know finding but wait conclusion some evidence
00:43:00
was found to suggest that ab eyes like the district approach
00:43:03
have a positive impact on these are time physical activity that
00:43:06
five districts regardless of the intent heidi oh that here
00:43:12
and so i think we really need to hold our journal editors are reviewers are arthur's ourself to higher standards
00:43:19
you know these people aren't selling something this is they're not selling chocolate or as uh not selling all of our
00:43:26
except they they're selling programs anymore i think these people just wanted a good they believe their programs work
00:43:32
then i work but i think that may lead to soak a white hat biases bias of wanting
00:43:37
to sell something because we've it's the right thing to do we've written about that here
00:43:41
and so this pen i can't go that this whole thing but basically shows the cycle
00:43:45
spin where what we can see is that really comes out through the offers
00:43:50
what authors don't include spin on their abstracts and university press releases don't include spin
00:43:55
in the press release the news media actually doesn't spend it that much
00:43:59
it's when the authors and the university press releases spin it that the news media spend so we have not the enemy in his office
00:44:07
our last section i'm gonna go into before i wrap profit fan or
00:44:13
okay i'm just gonna say a few things about my last get
00:44:18
oh i promise you wanna so it says what so your honours told me this morning i asked somebody said he six four five inches tall
00:44:24
and i asked them how tall is your side would you have us on how policies yes
00:44:28
is about six foot two inches tall that's interesting because a lot of people he said
00:44:32
well we know that height among farmers and scientists highly correlated
00:44:37
therefore they will tell you that my best guess of us on site if he's an adult is the height of before
00:44:43
but in fact that's not true because as friends call shows we have we question to them
00:44:48
the sounds of very tall fathers will be taller than average but they will be shorter on average than the forms
00:44:54
mice on whether shorter than average father is six foot one so that's regression to the mean alright um this
00:45:02
happens in people to so what happens if you take some kids were heavier than average at time one
00:45:09
and then you put them in a treatment and you follow them over time well
00:45:13
at time to there will be a little less heavy heavier than average
00:45:17
even if you are treatment was nothing at all that's cool regression to the mean
00:45:22
and by the other common thing we see is people run studies lack control groups
00:45:25
and then they think they found something it's really just regression e. r.
00:45:30
uh well i'm gonna huge the prime just skip ahead
00:45:38
to one thing where i show you the importance
00:45:42
of thinking mathematically so this randomised control trial came
00:45:46
out purporting to show that among chinese adults
00:45:51
who were given meridian massage for weight loss are and started out at about seventy five thilo's hundred
00:45:58
fifty ish pounds or so they lost nearly ten percent that their body weight in eight weeks
00:46:04
i thought this was a little bit extraordinary so i start to look at the numbers
00:46:08
and i notice that in the table they are the authors that put the
00:46:11
baseline be on my and the baseline way and then at the end point the
00:46:16
mean be your mind anyway well fine no be unwise kilograms over meter squared
00:46:22
i can solve a high if i know your mind wait now the mathematical people in the room are saying
00:46:27
wait a minute wait a minute the ratios means is not the mean of ratios and you're right
00:46:32
but if you use geometric means you can do some approximations and so we did and
00:46:36
what we're able to show is that in order for these results to be true
00:46:41
the average person in this study without the grown six centimetres in height in eight weeks
00:46:46
and so we're able to write to the journal and the office eventually
00:46:51
admitted that something was wrong you never told us what was wrong how this happened exactly and
00:46:56
and they put out in the table in which the weight loss is only happens big
00:47:00
so i think this is um you can look at these other slides you sell to rockwell
00:47:07
i think what we really needed to get people think more mathematically look at the numbers see if they make sense
00:47:13
is a great book about this it's fine it's funny you can listen to it on audio i highly encourage it
00:47:19
there's a a whole set of more high brow videos you can watch
00:47:24
from a meeting that we held at the national academy of sciences
00:47:28
on we produce ability we should back in march they're free europe or on the web now you
00:47:32
can go watch them using the p. analysis you wanted to talk about some of these issues
00:47:37
you can come to walk courses that we give in the summer on causal inference
00:47:40
or mathematical sciences funded by the n. i. h. if you wanna send your
00:47:44
students your post box and i think the sum of all say what i
00:47:50
think we really need is more training in statistics especially thinking about varies
00:47:55
really more training in it the discipline areas scale learning how to ask each other about our data
00:48:00
we need more use a professional statisticians not amateurs statisticians more training in mathematical thinking
00:48:07
of waiting for this recognition step this isn't of not the logic ad hoc agrees
00:48:12
development and use of checklists which sure that we do the right statistics
00:48:17
develop procedures in a culture that that supports expeditious and civil corrections or is detected
00:48:23
we together with the idea that well we found inner it's six crates in we have to be humiliated and you know refuse to
00:48:30
admit it and try to hide up just say you know what as long i made a mistake of sorry that's facts
00:48:37
and of course with these words from johns aqua so let's take these pat this path would users moral
00:48:43
with that are you blink and you're welcome visitors and we can talk some more than likely
00:48:57
because
00:49:01
just i yeah it actually
00:49:14
talk ah infrared here so long i it encourages us to to hire statisticians to is always
00:49:21
one question i have one shot operational tryouts show
00:49:25
uh we're really uh uh we randomised trials
00:49:28
well people or like to crossover charts 'cause effect radio some
00:49:33
of the issues of nor did it can you
00:49:37
what do you think that these trials i am very yeah that big a problem with corporations
00:49:42
to the meal and what i don't want to choose advantages this trial design may have
00:49:48
yeah people tend to favour crossover trials often in the nutrition literature when looking at short term effects like
00:49:54
what happens if i feed someone this at breakfast what are they don't one
00:49:57
sure what's there's friday or what's their media glucose rise or something
00:50:01
and they have a lot of nice features um person for person they can have a lot of increase statistical power
00:50:07
um but there are also some challenges so you have to deal with whatever facts so that's one of the things i don't like about cross overs
00:50:15
uh yeah if you get missing data to get some subject drop out it's a little more complicated
00:50:20
and if you want to anything adaptive whereby midway through the trial you say gee i i think
00:50:25
i wanna add some more subjects and you know to appropriate statistical fix up for that
00:50:30
it can be a lot slower more complicated in a crossover design so they're
00:50:34
both they're both good they're both advantages but they both the disadvantages
00:50:41
c. g. i. e. four discuss all for all
00:50:49
we're we're look at what gets shot was actually i a shop
00:50:54
and group averages track how much information i help
00:50:59
yeah yeah i know it's personalisation it was for a slap
00:51:04
how how how are we going to address this issue i'll
00:51:09
doing l. l. l. k. as well a strike
00:51:13
well yeah ha ha ha ha ha it's i think it's sure it's
00:51:22
i think uh on the benefit side maybe slightly easier to address i think they're a lot of different ways to go about it
00:51:28
i think one way to go about it is is just to say you know what i'm
00:51:33
gonna pick the low hanging fruit first let me find the big group of facts
00:51:37
you know yours is in the future first let me figure out what works on average that i'll try to
00:51:41
figure out when you want to buy the that's one potential response i think the second potential response
00:51:47
is to say okay let's go when an estimate where there are big
00:51:53
big variability we take on faith that there is a lot of heterogeneity response
00:51:57
you hear this for example f. t. a. hearings all the time
00:52:00
pharmaceutical company comes i'll show some day on weight loss for example and you see a histogram
00:52:06
some people lost a little weight maybe even gain some weight some people lost a lot of weight so you see big variability and somebody says
00:52:13
can we have more information about those non responders look at that variability response
00:52:18
and we always wanna say it you have no evidence that this variable response all using is variability change
00:52:24
if responsible change we wouldn't need the control group the reason we have the
00:52:28
control group is that change can be other things other than response
00:52:32
the standard clinical trials standard harold booklet control is not at all
00:52:36
well suited nor is a standard to prick also design
00:52:40
estimating variability response gotta go much more complex
00:52:44
design like of multiple perry crossover or
00:52:48
ballot design what we published a paper in art class genetics covers alternating
00:52:54
you know within individual you could do that you could estimate to the
00:52:57
individual their fax calls to estimate the variability the facts or time
00:53:02
and then eventually you could try to figure out predictors of which treatment worked better for which kind persecuted those kind of things
Conference Program
Nutrition advocacy in action: the politics of sugar vs. fat
Marion Nestle, New York University, USA
Sept. 28, 2017 · 9:27 a.m.
1337 views
Fat and carbohydrate recommendations - Have they changed?
Barbara Schneeman, University of California, Davis, USA
Sept. 28, 2017 · 10:14 a.m.
482 views
Promoting equitable health and wellbeing around the globe
Nancy Brown, American Heart Association, Dallas, USA
Sept. 28, 2017 · 11:30 a.m.
488 views
Gut hormones and nutrition
Sir Stephen Bloom, Imperial College London, England
Sept. 28, 2017 · 1:38 p.m.
1089 views
Nutrition, metabolic health, cancer and NCDs
Elio Riboli, Imperial College London, England
Sept. 28, 2017 · 2:27 p.m.
570 views
Infant and child nutrition and cognitive development
Wendy Oddy, Menzies Institute for Medical Research, University of Tasmania, Hobart, Australia
Sept. 28, 2017 · 3:51 p.m.
993 views
Microbes, metabolism and autoimmunity
Ramnik Xavier, Massachusetts General Hospital, Boston, USA
Sept. 29, 2017 · 9:06 a.m.
1955 views
Lipid metabolism in high fructose fed humans
Luc Tappy, University of Lausanne, Switzerland
Sept. 29, 2017 · 10:03 a.m.
1122 views
Food, health and disease: The evidence and reporting the evidence
Dennis Bier, Children's Nutrition Research Center, Baylor College of Medicine, Houston, USA
Sept. 29, 2017 · 11:11 a.m.
551 views
Deeper down the rabbit hole of data, analysis, and inference errors and suggestions for digging back out
David Allison, Indiana University School of Public Health, Bloomington, USA
Sept. 29, 2017 · 1 p.m.
501 views
Neural circuits of food intake - is it all about calories?
Harvey Grill, University of Pennsylvania, Philadelphia, USA
Sept. 29, 2017 · 1:55 p.m.
510 views
