Transcriptions
Note: this content has been automatically generated.
00:00:02
ah here i'm grateful for the opportunity so um the title of this talk to mitigate
00:00:08
recall i call deeper down the rabbit hole of data analysis and inference hours
00:00:14
and suggestions for getting back out and pour this idea of deeper down the rattles
00:00:18
reference of course the hours of wonderland falling down the rabbit hole work
00:00:22
we're up is down and black eyes widened is neither nothing is what it appears to be but also the idea going deeper
00:00:28
i knew what any was gonna talk about a little bit and i'm gonna try to build on some of his remarks
00:00:36
this omar disclosures i will read them all to you here you can read them yourself
00:00:41
and if anybody wants my slides just email me here i'll be glad to share them with you
00:00:47
so i'm gonna make some uh points about errors that occurred in the literature today
00:00:52
and i want to point out that we're all human and we all make mistakes and that includes maybe and so i'm
00:00:59
not just here to make fun of other people's hours um i wanna say you know what i make mistakes too
00:01:05
and as then he said earlier there's no shame in acknowledging
00:01:10
ones are is in fact quite the contrary once
00:01:13
you've been made aware of in or the the only shames to not acknowledge it and fix it
00:01:18
a few years ago my students got the thing i published a paper on b. m. i.
00:01:22
mortality in it we use a particular equation that was meant to correct self reported way
00:01:30
and um we showed up that using that
00:01:34
correction didn't make much difference in the formula for the corrections from another paper
00:01:39
and shortly after papers published we got you know you never wanna receive which is from the authors of
00:01:45
the paper was equation we used and they said we
00:01:48
cannot reproduce your results something's wrong or no
00:01:53
so okay now we're we're using an age data and so one of the nice things about it is
00:01:59
the data were all public so that's how they could show that they could we produce a result
00:02:04
so we said okay show us your code will show you our code we exchange code
00:02:09
eventually we're both able to figure out the mistakes it turned out both
00:02:13
groups have made mistakes that some mistakes the original paper in
00:02:16
in the transcription the equation we had some mistakes in setting up its use in our model
00:02:22
and so we work together remote uh a joint so rather more comment on
00:02:27
this me posted together and i think this points out several things
00:02:31
one is the value of raw data sharing the fact that
00:02:34
the raw data were out there the loudest expeditiously detectors
00:02:38
the second is that it didn't have the personal that we recriminations is
00:02:41
about the data that was about the code honest people made mistakes
00:02:46
and the third was we work together expeditiously to to correctly within state
00:02:50
look at the really embarrassing to correct it it's not that big a deal
00:02:54
don't really it overall change result important change to correct the scientific record
00:02:59
and so i wanna point all that out including and especially that i like to make mistakes and you know there's no shame in that
00:03:07
so some of you also know that we put out the obesity in energetic
00:03:10
so offerings every friday some of you probably got a few hours ago
00:03:14
if you don't hard on it and would like to be that
00:03:19
you know just uh you can look parrot that lincoln signed up or you can send me an email and i'll get you signed up
00:03:24
at three and every week we send out all about a hundred or so links
00:03:28
two papers in the literature describing something about obesity and energetic soon methodology and someone
00:03:35
and in order to put out that obviously we're looking at many more than
00:03:39
a hundred papers every week to pick what to put in there
00:03:43
and it's not uncommon that as i'm doing this every week papers come
00:03:47
across my screen and i look at this and i say
00:03:51
really could that possibly be true and then i'd go sometimes when i have time
00:03:56
i dig into little paper paper little more and i sometimes find now
00:04:02
it couldn't be true it's not true and we started noticing
00:04:05
a lot of these uh a couple years ago and
00:04:09
what we then did is we compiled all that we start writing some letters to the editor we got some papers vexed
00:04:15
and we actually had such an interesting collection of history with this that nature contacted
00:04:22
us and said we like to do a paper on this so we did
00:04:25
and so this paper came out which describes our experience
00:04:28
with not just detecting the errors which is part
00:04:32
of the issue i'll talk a little more about that for the big challenges is fixing the errors
00:04:38
and so unfortunately not all academics and novel journal editors
00:04:42
and here to the principles then you laid out in that he appears to an a. g. c. n.
00:04:47
which is if you find a mistake and it's really mistake admit it and fix it
00:04:51
and we have a lot of battles with waters and added and others
00:04:55
who don't want wood knowledge that a mistake is a mistake
00:05:00
so we'll talk about a few other kinds of common mistakes we see nutrition so that you can keep your
00:05:05
eyes out for them and so that you can help help think through how can we make this better
00:05:10
a denny's or we talk a lot about measurement so not to spend a great deal of time on and just to show you two slides
00:05:18
and here's the first one you've probably seen these maps of obesity levels by state odd nausea
00:05:23
so this is where i was from until very recently this is where i'm
00:05:27
from now um both places are places where there's lots of obesity
00:05:32
and people lot these maps they love rankings you know people like rankings in general and a lot these particular maps
00:05:39
and you have to always ask a good question sciences where the numbers come from where the data come from
00:05:45
and the way these where this confirms that the c. d. c. calls up a little more
00:05:49
than a hundred thousand people every year on the telephone and says to them among
00:05:54
other things but you're hiding much white and as you might not be surprised to learn
00:05:58
not everybody perfectly accurately reports their height and weight as nancy brown told us yesterday
00:06:04
so when you do that in order to do the state rankings you have to be making
00:06:09
the assumption that the differential reporting without the recording bias is constant across all states
00:06:16
and of course it's not exactly constant causal state so one of my body short
00:06:21
howard has actually measured haydn we data in many of these same state
00:06:26
and these are from the same point in time measured
00:06:29
writing wage on national represent what state representative data
00:06:33
and so you can see these different states and i just saw one
00:06:37
massachusetts down you think that's what you just talk last um n.
00:06:42
this is the ranking from the the self report data in this is the ranking from the
00:06:46
observe data and this breaking stays about the same which means that people in massachusetts are about
00:06:53
average honest compared to the rest of united states there no more or less honest then the rest of us
00:06:58
right you know now here is where sell them and it's a right down here and it goes way now
00:07:04
so it's just that at least with respect to hide away we don't about other things but with respect highway
00:07:09
people know about uh probably a little bit more honest than people in the rest of the country
00:07:13
and then if you look at others that go way up you know a year is missouri
00:07:17
you know it's quite the contrary probably a lot less honest then people the rest nice states
00:07:22
now you might say you know what who really cares about state make use of high to
00:07:27
make does it matter and i'm not sure does i'm not sure we should care
00:07:31
but if we spend as much attention as we do on it if it's important not to publish and talk about mouse will get right
00:07:38
and it's interesting i was talking to somebody on where your comments was talking to me about the hague suppose on
00:07:43
so we've got you know some scientists around the world are using these high tech machinery to figure out whether
00:07:49
they expose on exists and here is in two thousand seventeen we can't figure out a way people
00:07:55
so it seems to me that we we ought to really hold our
00:07:57
science too liberal higher level that we wanna be taken seriously scientists
00:08:05
alright maybe a little more precision though now the other thing is about support denny did a great job with this
00:08:11
one not gonna be labour i just wanna are point that we have a paper on this our own
00:08:16
comments of the mainly about almost exclusive about energy measurement not
00:08:21
i'll mention results work of food intake by sort
00:08:23
of work but i think here's this three point don't wanna make because we hear this a lot
00:08:28
and the key point is that just because the measurement method one has a hand is the best available does it make it adequate
00:08:35
so we hear a lot people say what david you haven't proposed anything that
00:08:39
what else am i gonna do i cannot use doubly label water in
00:08:43
twenty thousand people action yeah maybe that could be true you might not
00:08:47
be able to and they say it's all important that i study
00:08:52
this aspect of energy bounce in a large apathy melodic study it's really interesting questions yeah it is
00:08:59
and they say so the best thing i have is this that that you says invalid yeah that's true
00:09:04
and they say so well should i do well then uses that maybe just not ready to study
00:09:09
right if i told you i the best available space shuttle to take you to morris
00:09:14
and i could convince really was the best and i say but by the way it blows up ninety five percent of the time
00:09:21
do you wanna get on that special maybe so maybe i'm just not ready for the trip to mars yeah
00:09:26
maybe that's important scientific questions a lot we can learn by landing on mars but maybe i'm not ready to go yet
00:09:34
alright errors of design i think is another problem again danny talk about the observation start
00:09:39
i implore the thing about observation all at the genealogy is not that we should do it
00:09:44
but it's sometimes the over use or over interpretation of it and part of that is not knowing when to stop
00:09:52
so this is the quotation from daniel common nobel prize winning economists from his book thinking fast and slow
00:09:58
any says reliable way to make people believe in fall so does frequent repetition because familiarity
00:10:03
is not easily distinguished from truth authoritarian institutions of marketers have always known this
00:10:10
you do not to repeat the entire state of the fact we're idea to make it pure true
00:10:16
people will read who repeatedly exposed for example to the phrase the body temperature the check in
00:10:23
well more likely to accept this through the statement that the body temperature chicken is one hundred forty four degrees
00:10:30
by the way i look this up on the u. s. d. a. website much of coke at the body temperature which it is not one hundred forty four degrees
00:10:37
ah but the more you say at the more people tend to accept that is true
00:10:42
think about some of the many things you've heard over and over in time up until about a year or so
00:10:47
ago when we were all the many talks about got microbe iota the typical way to introduce such a talk
00:10:53
was to say there are trillions of cells of microbes in our
00:10:58
body they outnumber us by one to two orders of magnitude
00:11:03
there are approximately ten percent of our body mass is microbes and then yeah so people said
00:11:09
it all over again they didn't decide a reference anymore where did this come from
00:11:13
finally somebody go back in the nineteen seventies paper on the diary a sample one sick man
00:11:19
you know and and so you guys think we we hear these overall we to the point we forget to ask
00:11:26
so here's an interesting thing in the early nineteen nineties somebody did a study first one we
00:11:32
not in which they correlated in epidemiological observation study
00:11:37
the association between breakfast skipping versus eating
00:11:42
and obesity and what they found was that if you skip breakfast you were more likely to be
00:11:49
overweight or obese then if you did it relative risk about one point seven five or so
00:11:55
that's fine good study any might say at that point we need to replicate this let's
00:12:00
see the holds up very good idea so people came along the replicated it
00:12:05
and what and we round up all at the time a post doc with me did was he took all these ended
00:12:11
and accumulating that an else's mother would isn't that allies in the data as well what's occurring in real time
00:12:17
and saying how with the point estimate changing and how are competent changing
00:12:22
what you can see over time or more data start looking like
00:12:26
a point estimate starts changing parameters or we're still hovers around one point seven five
00:12:31
the confidence intervals gets smaller and smaller with significant here at this point
00:12:36
the p. about you use a lot of negative log p. value
00:12:40
it's already at this point at about ten to the minus three and so what you think about
00:12:48
here is that um we already knew conclusively that it was an association nearly nineteen nineties
00:12:55
at that point it was no more need to replicate vision epidemiological research the answer was there
00:13:00
we get right skipping breakfast is associated basically got it stopped no more study is needed
00:13:06
how often do we see at the end of a paper we always see and in conclusion more research is needed
00:13:12
when there's some whatever any conclusion no more research is needed on this particular question
00:13:18
that's really no more research was needed up at the logic observation nature by the end of
00:13:24
the nineteen nineties on the question of whether breakfast and something was a source of obesity
00:13:29
what was needed was randomised control trials to see if it was causal and yeah what happen is we drove that
00:13:36
and uh all down until the p. value when we start counting was ten to the minus forty two
00:13:41
right holding the inverse of adding cans number for the number protons in
00:13:45
the universe this is a little more precision then we need it
00:13:48
and it was not a good use of of our time it was not a
00:13:51
good use of our resources and more importantly wasn't getting the question answered
00:13:56
but what did it do by the mere exposure fact that continent talked about
00:14:01
we devoted our pages are time or resources to research that increase belief
00:14:07
without increasing knowledge and so to the point where people came
00:14:11
to believe that wasn't on the politically demonstrated fact
00:14:16
that eating versus skipping breakfast like the better way control
00:14:19
surgeon general had on her website until recently
00:14:24
alright now what the observation research we heard some good talks on that yesterday
00:14:30
uh one point one speaker said we've controlled for all possible can founders
00:14:36
i don't know how you control for possible to downers except the ram station that's one that i know that controls for possible confound
00:14:42
but the dialogue we're gonna do we believe the observation that pretty
00:14:46
much every so often goes we recognise that there the limits
00:14:51
the author then says yes there measurement problems yes that's
00:14:55
possible confounding yes there are other potential issues but
00:14:59
let me tell you about how good my study was it was a very homogeneous
00:15:03
sample they were all from uh switzerland and all you know very homogeneous
00:15:09
and they were all nurses so they all have the same socioeconomic
00:15:12
status and i measure things very carefully and so on
00:15:15
and think oh okay maybe that should make me believe it so i thought what
00:15:19
we took this to the limit what if i did that the near perfect
00:15:23
observation let the noise sixty so i'm gonna get a lot of identical twins or maybe
00:15:29
a genetically identical so all this concern about kinetic energy is gonna be gone
00:15:35
and i'm gonna get away from denny's concerns about measurement error i'm not gonna ask them what the
00:15:40
i'm going up observe them twenty four hours a day and measure everything that goes into now
00:15:45
and i'm gonna get rid of the socioeconomic status concerns because somebody put them all in the same housing you know
00:15:51
adam points out that sci stripes these things pretty quick and found a good point
00:15:56
we put them on the same housing will give them all the saying opportunities for fruit and
00:16:00
someone and then we're gonna see what happens what in that situation could i'd hop on
00:16:05
that my observation all epidemiological study done better than you could ever do
00:16:11
would give me the same answer that are randomised experiment would give me and i realised i've actually done this study
00:16:17
i've just done it in mike's nineteen people so one of our mouse studies we have some my spare reading i love
00:16:23
human some ice they're fed a little less and some ice the randomly assigned to be fed a little less still
00:16:29
and within the ad would be improving actually also the other groups there's some variation
00:16:32
how much the mice shoes to eat so now i've got two elements
00:16:36
i've got an attribute grouper mice shoes to you more or less spontaneously and
00:16:42
i've got an element were i randomly assign my steve more or less
00:16:46
and so i can carly how much the mouse choose to you without comes and
00:16:50
then i can look at the experimental effect of assigning mice to different amounts
00:16:55
and here's the experiment or fact this is been shown many times if
00:16:59
you freedom of moore's ever always we let them go beside where
00:17:03
they don't live as long as if you colour to restrict them or make them lose some weight so no big surprise there okay
00:17:09
but here's the association of self selected energy intake in
00:17:14
these different groups with longevity or like spam
00:17:18
it's the exact opposite and statistically significant in both directions
00:17:23
so that if i were to do this as an observation of starting with
00:17:26
the best measure menu could ever achieve better measure the garbage even humans
00:17:31
i would conclude that eating more causes you
00:17:36
to live longer but if i'd say what is the effect of making animals
00:17:41
the last i say eating less causes you to you long live longer
00:17:46
and what i think the shows is that there is something fundamentally different about
00:17:51
being randomly assigned to things versus self selecting even when you tightened up your measurement and
00:17:57
got genetically homogeneous group rides all inbred mice the role uh in probability genetically identical
00:18:04
and so i think it really suggest that there is no perfect observation started i guarantee matt
00:18:09
how well you do it that you get the true estimate of the causal effect
00:18:14
some people have said was what this just probably means david is that
00:18:17
these animals that choose to eat more to hold your animals
00:18:21
the cigarette animals choose the lesson that to try and it's amazing yeah
00:18:26
that's the point that is a confounding factor which is not
00:18:30
experiment with the lights are also errors of analysis
00:18:37
one of these occurs in that analysis some other analyses are as in remote can once
00:18:41
said that analyses are easy to do unless you're doing it well that's hard
00:18:47
so that analyses have become this this quote unquote cheap way to get a publication the software is out there
00:18:55
anybody can go in and type in their effect sizes or the numbers and the software will dutifully
00:19:00
spit out stuff but the problem is first you have to know what the type then
00:19:04
and in particular you have to know which variances to include and that seems to confuse a lot of people
00:19:10
means most people get right variance is not so much
00:19:15
this is a paper that came out long time ago twenty years ago now
00:19:20
and we came across my desk um actually we're responding to a paper by these guys out when
00:19:25
it came across my desk i know something about the effect size in this uh thirty
00:19:33
this net analysis was reported as a whole wednesday standardise mean different one point nine six
00:19:40
and remembered something uh when this book labels and that any also signed distributions are standardise effect sizes rising
00:19:46
maternal slogan domains also standard research it is unusual for the magnitude to be as big as one
00:19:52
why where these big one point four and extraordinary would be as big as to
00:19:58
so now we say not only is one study got something almost as big as to what the average across an
00:20:04
entire body of literature of hypnosis for weight loss is as
00:20:08
big as to just really could that be true
00:20:12
so there weren't that many papers so i said the mouse fate with my post archetype cole porter papers out
00:20:18
so people the more on i said let's recalculated not that much work was it that doesn't papers all
00:20:24
we recalculated we found that there were errors in more than half the paper calculations
00:20:29
we fix the errors and the one point nine six in down to point two six
00:20:35
and if you actually throughout a study that was look quite questionable
00:20:40
um yeah actually came down to point to one wasn't even statistically significant anymore
00:20:45
so what we see is people um seem to make lots of mistakes and not an l. c. so
00:20:49
take note analysis with a grain of salt please don't ask you graduate student to do one unless he or she is a statistician
00:20:56
please have a professional statistician involving human announces just because your students as they can
00:21:00
turn the software on it three doesn't mean they know what they're doing
00:21:04
um any reading that analyses look at the effect size is a lot of times we pick these up just 'cause i
00:21:09
looked margo mm one of them's already over here what's that one doing over there we looking we find mistakes
00:21:15
that's exactly what we've done here one came unglued commanding recently took a man and helps with weight loss
00:21:22
and again i looked at it held up this little graph there about a half dozen effect sizes
00:21:26
one of them was way off on the right i said to my latest post doctor shawna
00:21:31
colds b. s. additional it'll pull the paper down let's look at it again she
00:21:34
came back she said i can we calculate five but not the six
00:21:38
so we contacted the original authors they said well we have some original data
00:21:44
from the first daughters who had the paper we met our wise
00:21:47
and a simple we send it to us they got permission to do with the centre was
00:21:50
really original data it was just calculations of means in estes looking at these calculations
00:21:56
and i couldn't figure out what was what so sent the to kong whatever starters the parking you figure this out
00:22:01
so let me go work out of what comes back and on our any says doctor asking me think i'm crazy
00:22:06
i see okay he says but take a look at this and he says you see other calculate the mean
00:22:11
difference here you know mean it three minus mean posters yeah it's just a look at the standard deviations
00:22:18
and he said if you ignore the signs it's the same as the
00:22:21
standard deviation three minus this innovation pose is it yeah oh
00:22:27
and he said but you can't even make it stand aviation i should know unless
00:22:30
we work with imaginary numbers here your nose like a square root right
00:22:34
and he said i think they just drop the sign which is with packed exactly what they did so they
00:22:39
didn't know how to calculate variances they thought that difference isn't stand deviations was the standard deviation of different
00:22:46
screwed up the results we help them fix it again it's very nice they admitted they made a mistake
00:22:50
we work together we published a little note together fixing it and what was significant is no longer
00:22:55
significant so good command in is not there was not significant evidence that it causes weight loss
00:23:03
this is one of my my favourite ones lately this is the darling design art
00:23:09
public health um people wanna do community into branches school based interventions and so on
00:23:15
if you're going to school for example when you say we're going
00:23:18
to um give a fruit and vegetable promotion program to kids
00:23:24
it's hard to go say are you get the fruit or vegetables and you don't need any future vegetables that's a real strange
00:23:30
so what usually wind up doing is you you take an entire school and you say we randomised this whole
00:23:35
school to let's say get fruits and vegetables instructions about fruits and vegetables or physical activity whatever it is
00:23:41
we go to the next school they get randomised to something difference all through randomised at
00:23:45
the level school work county or city or building for classroom or physicians cactus
00:23:53
and the mistake people often make is that the forget that they
00:23:57
randomised at the level is think will cost school the office
00:24:02
and then each week the individual subject individual person as the unit of analysis and
00:24:08
if you do that your statistics are as they say in yiddish for cocktail
00:24:13
um you can figure out what that means so um
00:24:18
i'm not gonna go through this but we published a paper there's the reference in ages yeah
00:24:22
and talking about cluster randomised trials and the special things that need to be done
00:24:28
and here's an interesting example so um this is a effects with twenty month
00:24:32
cluster randomised school based intervention trial on b. m. isolate boys and girls
00:24:37
and the authors wrote as only two percent of the variance and be on my waist circumference
00:24:43
was explained by group we did not adjust for clustering analysis
00:24:47
now to set you've gotta would just close to new houses you don't do it's not down they said well oh so small
00:24:53
the intro class correlation was only pointed to two percent it's so
00:24:57
don't need to worry about and the side a particular reference to support reference but i've we've got about second fido
00:25:05
what's interesting is if you go back to that weapons you actually read it it directly contradicts
00:25:11
them it says even small into class correlation coefficients as small as point o. two
00:25:16
can really screw up your results can we still simulation of this and in fact we show that
00:25:23
with their sample sizes and so on this is the we are estimated type one o. eight
00:25:28
if you ignore the intro class you can order clustering as they did with this level of intro class correlation
00:25:33
right so instead of having a point of all level they were really operating
00:25:37
at somewhere between the point one all and point one five level
00:25:41
and so many of the things that you readers cluster randomised trial results
00:25:45
or in fact just plain wrong many the things that are purported to be
00:25:49
significant or not this is a big problem again if your journal letter
00:25:53
i would ask you keep a look out for ice services so see that are for a. g. c. and for i. g. all for obesity
00:25:59
and on all three of those journals it so just like a standing or they see
00:26:03
the work last they just you send it to me were the other statistical editors
00:26:07
just to to take care of it otherwise we get crazy result about one for good time
00:26:12
right here's an example of what happens when you don't send it to the statistic letters first
00:26:17
so this was a cluster randomised trial of of gardening intervention that was
00:26:21
probably still be city that had to be retracted because of
00:26:24
the analyses were done correctly in the results didn't hold up
00:26:31
um cluster randomised trials and attractions come together to be particularly sticky
00:26:36
this is an interesting story up paper came out from
00:26:39
a brazilian group energy a journal called obesity fast
00:26:44
and this was um a methodology paper telling people about some
00:26:47
of them not the logic issues in cluster randomised trials
00:26:50
and what they should be concerned about how to do certain things and as i read it i thought gee i don't think this is right
00:26:57
and not only do i not think it's right at some simulations and some men analyses in a make a point
00:27:02
is it i don't think you need the simulations are mad analysis for that i think this is noble
00:27:07
from first principles mathematically this is just that should be obvious to any trends that station
00:27:13
and so i sent around so my buddies who are you know experts in cluster minimised process it
00:27:18
you think i've got this right this really yeah yeah you gotta write this paper that make any sense i said
00:27:23
okay so let's write a letter to the editor so we wrote a letter to the and we said
00:27:28
support the look a cluster randomised trials but respectfully the statements made
00:27:32
by the authors are in contradiction to mathematically provable facts
00:27:36
and that unknown in literature and we really think there for your to retract the paper because is putting out misinformation
00:27:44
and the latter took our paper sent it to a
00:27:48
another statistician independently boss who came back to us
00:27:52
these guys in alabama maybe crazy but they know what they're talking about
00:27:56
on this particular issue and um the paper should be attracted
00:28:00
and sent it to the authors and your preset no we don't want retract our paper
00:28:06
and the other said but you're wrong should be retracted and they should
00:28:10
know and the other was a forty track but chose not to
00:28:13
interestingly the other then wrote a whole editorial describing this a retraction watches little story about
00:28:19
the whole thing here so i think this points out to really interesting things
00:28:24
one is how confusing cluster randomised trials or to people that
00:28:27
even mythology still confused about them and the second is
00:28:31
the problems we have with the journals where they'll really wanna
00:28:34
take responsibility authors and editors seem unwilling to take responsibility
00:28:40
you know if you hang around in places like alabama you might hear about singer named eric church
00:28:44
who's gonna popular songs as i learned that from a three year old and refrain is
00:28:49
when you're wrong you should just say so i learned that from a three year old so i think we need to
00:28:54
get our editors in ourselves in a little bit harder just being will say you know well mainly staple wrong sorry
00:28:59
all the paper correct the paper it you know it's better to correct the record uh they're not
00:29:07
i and number one of the problems we have is what we call the dunes are doing
00:29:10
this is an acronym one of our our guys made up for difference in nominal significance
00:29:16
we see this a lot in randomised trials often
00:29:19
small randomised drastically exact dietary supplements nutrients
00:29:24
um and somebody doesn't get a statistically significant result
00:29:29
when they compare the two parallel randomised groups
00:29:34
or maybe they just don't know do that test and so instead what they
00:29:38
do is they first cast whether the treatment group change significantly from baseline
00:29:43
and then they cast whether the control group change significantly from baseline and if
00:29:48
the treatment group change significantly from baseline let's say please point four nine
00:29:53
and the control group didn't let's say p. was point o. five one
00:29:57
they say ah this group change this group didn't i
00:30:02
have an effect it's significant i've shown advocacy
00:30:05
and most of you immediately realise that that's not about task if you do the math of it you relies wonder
00:30:12
the worst case scenarios with equal groups um you have you can have a fifty percent type whatever rate
00:30:18
in that situation i would on equal groups you can get the type one error rate because two hundred percent
00:30:24
um martin planned and doggone wrote a paper in a. g. c. and
00:30:29
describing this if you wanna have a detailed discussion of this
00:30:32
your other papers and letters well and here's another paper that pointed out
00:30:37
it was these guys put up april flax seed they show that
00:30:40
flax seed seem to have beneficial effects on cardiovascular rest and again
00:30:45
as many c. of looking at it said gee this
00:30:48
doesn't seem right seems like they made the denser and then i know says
00:30:52
little windy said we have poster or data here publicly which i'd martin
00:30:57
for so we picked up the raw data we analysed it i'm looking at the correct between groups that it was not even a with
00:31:04
of an effect and so we wrote to the other and then here both the other in these people did
00:31:10
the right thing and they said you know what we're sorry we names they would use the wrong analysis
00:31:15
and the paper was retracted there's another one where this came up recently
00:31:21
this paper just came out pediatrics in there's a letter correction
00:31:24
percolating through pediatrics system now so this be corrected um
00:31:30
i first these data theme okay since the trial net mormon it'll
00:31:33
be sitting in in you blow up repute roberta pupil children
00:31:38
and what you see is easy these different weight losses in these different groups possible met one and keep you look you blue
00:31:45
and here's the difference in different so in the creek you will group
00:31:51
the net forming group last well point two more cagey that could
00:31:56
use without numbers um then uh then the placebo group
00:32:00
and the same thing occur in the pure will just okay that's fine consistency
00:32:06
reach last point two cages more and then the statement comes out
00:32:11
that formant increase the b. m. i. z. score improved inflammatory choreographed relate obesity parameters
00:32:17
in pretty people told the banana pupil talking hands the differential response according
00:32:22
to puberty might be related to the dose of my form
00:32:25
so i imagine you're talking to a clinician and use arrives import paper that just came out you have to
00:32:29
think about the differential response to my foreman if you're dosing or preview we'll versus people to others
00:32:36
oh thank you professor hobbies that differential sparked well me tell you if you know what i would do or zero
00:32:46
one the differential response that doesn't exist well because what they did is they basically said well we got a significant
00:32:52
result in one group and not the other and then drew this crazy conclusion and so that's been corrected now
00:33:00
one of the other things that i see in the epidemiological uh literature and sometimes in the i'm allergic to is relying on
00:33:06
statistical ad hoc reason i think it's important to uh be
00:33:10
on the look out for the distinction between assertion improve
00:33:14
and so some of the things he's seen the epidemiological traffic a lot eliminating really getting
00:33:19
the uh my mortality you know he's wishing papers on on if you're interested
00:33:23
eliminating biologically um implausible values in child waiting hours he's
00:33:29
and eliminating implausible food intake analyses multi supported intake so because
00:33:34
we were talk a lot about supporting take your thought
00:33:37
one might mention this one so comments thought is to say well i know that the
00:33:41
salt purported energy intake so not so good and people turned on the report
00:33:46
so i'll use some rules for the finding who's really implausible on the
00:33:50
reporter and is a rule culpable bird will that some people use
00:33:54
and one side to find this subset where their voices also
00:33:57
really implausible i'll throw them out of my analysis
00:34:01
now redo my analysis and the presumption is now my
00:34:04
new analysis will get results better better more closer
00:34:08
to the truth then the original analyses and so then i can be more confident than those results
00:34:14
and the question is is that true and it's interesting 'cause you say well i sort of have some
00:34:18
intuitive appeal to it but where's the proof of this is their model can someone right out
00:34:23
some parameters for me in some equations that show that parameter estimates get closer to their
00:34:29
true value when this is done them when it's not done and i haven't seen anything like that
00:34:34
so we just did a little analysis where we took some real data where we had
00:34:38
uh objective measurements of energy intake by doubly above water rip stop report measurements of energy intake
00:34:44
and then we had some um we're blood pressure measurements just to look for something uh correlated with
00:34:50
and here's the slope of the regression of blood pressure on energy intake when you
00:34:54
use energy intake by double double whatever object that's about point six nine
00:35:00
here it is when you stop work it's about one two three
00:35:03
to four when you eliminate subjects by this goal were rule
00:35:07
it actually gets worse not better now i'm not saying that's gonna happen in every case what happened in this case
00:35:13
and what it really shows is that we don't know what can happen unless someone really writes this now
00:35:18
so i think a real thing we need to be um on the look out for these
00:35:21
kind of statistical ad hoc agrees that people use particularly in observation epidemiology with the state
00:35:27
here's this thing on throwing out there ought to make my data better without approve that actually has particular statistical properties
00:35:37
and the general question for me that i think is uppermost but his little joke
00:35:41
under certain phones the statistical consulting department to form them on doing the study and i'd rather just do my
00:35:47
own statistics so i don't need your help i just wonder if you can suggest a good statistics text
00:35:53
and the consulting statistician says i'm so like to call the voice was gonna do brain surgery can you suggest a good text on that
00:36:01
and i think we really have to think about i won't ask for short hands on this
00:36:05
but you know i bet a lot of people in this room are trained in physiology
00:36:09
in nutrition in psychology can you see ali g. and a few other things like that
00:36:14
where we all got our one warm may be to statistics courses in graduate school
00:36:20
and then a few years later we were getting a rusty on the software
00:36:25
and our up and coming grad which do not post doc was getting good at it and here he knew had turn as p. s. s. on
00:36:32
and you know anybody could turn s. p. s. is on it some ice you pointing you click in it with all these tables and so we think you're
00:36:39
the kid knows what he or she is doing they say they know what they're doing so i get the data the kid
00:36:44
i said allies it's a cage do a good job i get professor okay good and that's the quality control checks
00:36:51
and i think i've done that right i i was a kid sometimes them the professor and
00:36:57
i think that may then maybe in it was a day when i was
00:37:00
okay and i still think that is the norm in many many environments
00:37:05
i don't think that should be the norm when we really care about the regular for dayton will drop policy on it and we're gonna
00:37:11
you know have advocacy based on events or by hours of recording so
00:37:20
here's you know something i pointed out yesterday the facts matt
00:37:23
um assertions are interesting but the facts matter
00:37:28
this is from a letter more coke and i wrote in response to a
00:37:32
paper that were this title in john back in two thousand ten
00:37:36
and a group are really cool paper actually about phantoms good paper still is a good paper
00:37:41
and in this paper which they quantified how often people use spin to find is kind of
00:37:45
distorting were exaggerating results a little bit yeah you didn't get
00:37:49
a statistically significant fact in your randomised control trial
00:37:53
they made a statement they said our results are consistent with the idea
00:37:58
something like this that um financial conflicts of interest bias results
00:38:04
will lead to more spam something ones i so that doesn't sound good i mean to look at these data
00:38:09
so when i i could not find anywhere in this table in the paper
00:38:15
the data supporting yeah so i email my from market to market and maybe i'm going
00:38:19
blind can you find this thing is i don't see it anywhere the pink
00:38:23
so we wrote a letter to a jam we said this is an interesting assertion we can't find the
00:38:28
data you know the authors say they collected data on whether this is industry sponsored or not
00:38:34
but we don't see in the paper could they i could they show was the result the office came back and said
00:38:40
thank you for this comment our statement was an overstatement there was no significant association between uh
00:38:48
funding source and you suspect so not only was the low significant association
00:38:56
between funding source you spent by the fact that the authors
00:38:59
had spun the authors had stated that there was incorrectly this is a paper that just came out a few months ago
00:39:07
and this with uh a paper looking at discrepancies between the
00:39:12
major endpoint in a randomised control trial that'll be city
00:39:15
versus what was registered so so for example somebody registers the trials as my main endpoint is body fat
00:39:22
and then the paper comes out and they say our primary endpoint is wasted conference see something like
00:39:28
that i think um maybe they didn't get a good result them by fatty switched away circumference
00:39:34
and it's okay where you've wasted conference but just tell people you switched you know just be honest be transparent is um
00:39:42
so we looked into this nice again this is interesting because the state was the industry funded studies
00:39:48
were the most likely to have a major discrepancy with this is that it's done some
00:39:52
different history and we go look at these data and we'll look at the table
00:39:56
i thought it looks to me like the number for industry lower than any other group
00:40:02
then again i showed some my friends i said am i going blind them i confused him upside down
00:40:08
have i gone for the rabbit hole and they said oh no not as far as we can
00:40:12
tell it looks like industry funded studies all the least likely studies have a discrepancy between
00:40:20
the registration and the publication so we wrote a header and the authors have come back and said
00:40:26
alice in college or correct our steak was exactly opposite to what each of them we apologise
00:40:32
so we got a really these things gotta look at the data don't take everybody's word for
00:40:36
it when they say it is um incontrovertible that funding sources associate with this or that
00:40:45
our headlines i want the labour it because i know we we all know the disconnect between the headline
00:40:51
in the study is often crazy it's just examples which might ask where is this come from
00:40:57
well some of it is an hour observation all studies this is an a. g. c. and little little
00:41:02
old i think maybe better now a. g. c. n. i. d. o. b. c. journal nutrition
00:41:06
this was the frequency of studies finding an association and reporting it as
00:41:11
causation and you can see in one journals higher than fifty percent
00:41:17
talk about spending his non industry funded studies from the c. d. c.
00:41:23
um effect of healthy schools problem uh problems of overweight obese in california
00:41:28
the press release dates now is the aligned celebrate a ten year anniversary in new period you study confirms
00:41:34
we are delivering on our mission of reducing the prevalence of childhood obesity
00:41:39
an important means of supporting schools are reducing obesity that sounds really good well people be testing is we're delivering
00:41:46
what the data for this years from start analyse showed no difference between
00:41:51
healthy school problems causing control schools in overweight and obesity problems
00:41:58
further down the rabbit hole
00:42:01
and yet they say how peaceful probably appeared report means of supporting schools are reducing obesity
00:42:07
there's no connection between the data and the conclusions coming out of the c. d. c.
00:42:14
here's another one the impacted area based initiatives on his collectively trends in deprived areas
00:42:21
i think the prime target district showed significantly part of changing walking trend
00:42:27
um and intervention the trend uh deprive audio physically larger compared to the
00:42:32
rest no i haven't but not compared to other deprive disputes
00:42:35
for cycling in sports neither deprived just it's not controlled just showed significant trend in change
00:42:40
for all the time for selectively outcomes trend changes when
00:42:43
not related need tentative environmental interventions think well that's
00:42:47
a lotta now is loud nothing going on here right this violates pennies principle that something must happen
00:42:54
it's okay sometimes you gotta know finding but wait conclusion some evidence
00:43:00
was found to suggest that ab eyes like the district approach
00:43:03
have a positive impact on these are time physical activity that
00:43:06
five districts regardless of the intent heidi oh that here
00:43:12
and so i think we really need to hold our journal editors are reviewers are arthur's ourself to higher standards
00:43:19
you know these people aren't selling something this is they're not selling chocolate or as uh not selling all of our
00:43:26
except they they're selling programs anymore i think these people just wanted a good they believe their programs work
00:43:32
then i work but i think that may lead to soak a white hat biases bias of wanting
00:43:37
to sell something because we've it's the right thing to do we've written about that here
00:43:41
and so this pen i can't go that this whole thing but basically shows the cycle
00:43:45
spin where what we can see is that really comes out through the offers
00:43:50
what authors don't include spin on their abstracts and university press releases don't include spin
00:43:55
in the press release the news media actually doesn't spend it that much
00:43:59
it's when the authors and the university press releases spin it that the news media spend so we have not the enemy in his office
00:44:07
our last section i'm gonna go into before i wrap profit fan or
00:44:13
okay i'm just gonna say a few things about my last get
00:44:18
oh i promise you wanna so it says what so your honours told me this morning i asked somebody said he six four five inches tall
00:44:24
and i asked them how tall is your side would you have us on how policies yes
00:44:28
is about six foot two inches tall that's interesting because a lot of people he said
00:44:32
well we know that height among farmers and scientists highly correlated
00:44:37
therefore they will tell you that my best guess of us on site if he's an adult is the height of before
00:44:43
but in fact that's not true because as friends call shows we have we question to them
00:44:48
the sounds of very tall fathers will be taller than average but they will be shorter on average than the forms
00:44:54
mice on whether shorter than average father is six foot one so that's regression to the mean alright um this
00:45:02
happens in people to so what happens if you take some kids were heavier than average at time one
00:45:09
and then you put them in a treatment and you follow them over time well
00:45:13
at time to there will be a little less heavy heavier than average
00:45:17
even if you are treatment was nothing at all that's cool regression to the mean
00:45:22
and by the other common thing we see is people run studies lack control groups
00:45:25
and then they think they found something it's really just regression e. r.
00:45:30
uh well i'm gonna huge the prime just skip ahead
00:45:38
to one thing where i show you the importance
00:45:42
of thinking mathematically so this randomised control trial came
00:45:46
out purporting to show that among chinese adults
00:45:51
who were given meridian massage for weight loss are and started out at about seventy five thilo's hundred
00:45:58
fifty ish pounds or so they lost nearly ten percent that their body weight in eight weeks
00:46:04
i thought this was a little bit extraordinary so i start to look at the numbers
00:46:08
and i notice that in the table they are the authors that put the
00:46:11
baseline be on my and the baseline way and then at the end point the
00:46:16
mean be your mind anyway well fine no be unwise kilograms over meter squared
00:46:22
i can solve a high if i know your mind wait now the mathematical people in the room are saying
00:46:27
wait a minute wait a minute the ratios means is not the mean of ratios and you're right
00:46:32
but if you use geometric means you can do some approximations and so we did and
00:46:36
what we're able to show is that in order for these results to be true
00:46:41
the average person in this study without the grown six centimetres in height in eight weeks
00:46:46
and so we're able to write to the journal and the office eventually
00:46:51
admitted that something was wrong you never told us what was wrong how this happened exactly and
00:46:56
and they put out in the table in which the weight loss is only happens big
00:47:00
so i think this is um you can look at these other slides you sell to rockwell
00:47:07
i think what we really needed to get people think more mathematically look at the numbers see if they make sense
00:47:13
is a great book about this it's fine it's funny you can listen to it on audio i highly encourage it
00:47:19
there's a a whole set of more high brow videos you can watch
00:47:24
from a meeting that we held at the national academy of sciences
00:47:28
on we produce ability we should back in march they're free europe or on the web now you
00:47:32
can go watch them using the p. analysis you wanted to talk about some of these issues
00:47:37
you can come to walk courses that we give in the summer on causal inference
00:47:40
or mathematical sciences funded by the n. i. h. if you wanna send your
00:47:44
students your post box and i think the sum of all say what i
00:47:50
think we really need is more training in statistics especially thinking about varies
00:47:55
really more training in it the discipline areas scale learning how to ask each other about our data
00:48:00
we need more use a professional statisticians not amateurs statisticians more training in mathematical thinking
00:48:07
of waiting for this recognition step this isn't of not the logic ad hoc agrees
00:48:12
development and use of checklists which sure that we do the right statistics
00:48:17
develop procedures in a culture that that supports expeditious and civil corrections or is detected
00:48:23
we together with the idea that well we found inner it's six crates in we have to be humiliated and you know refuse to
00:48:30
admit it and try to hide up just say you know what as long i made a mistake of sorry that's facts
00:48:37
and of course with these words from johns aqua so let's take these pat this path would users moral
00:48:43
with that are you blink and you're welcome visitors and we can talk some more than likely
00:49:01
just i yeah it actually
00:49:14
talk ah infrared here so long i it encourages us to to hire statisticians to is always
00:49:21
one question i have one shot operational tryouts show
00:49:25
uh we're really uh uh we randomised trials
00:49:28
well people or like to crossover charts 'cause effect radio some
00:49:33
of the issues of nor did it can you
00:49:37
what do you think that these trials i am very yeah that big a problem with corporations
00:49:42
to the meal and what i don't want to choose advantages this trial design may have
00:49:48
yeah people tend to favour crossover trials often in the nutrition literature when looking at short term effects like
00:49:54
what happens if i feed someone this at breakfast what are they don't one
00:49:57
sure what's there's friday or what's their media glucose rise or something
00:50:01
and they have a lot of nice features um person for person they can have a lot of increase statistical power
00:50:07
um but there are also some challenges so you have to deal with whatever facts so that's one of the things i don't like about cross overs
00:50:15
uh yeah if you get missing data to get some subject drop out it's a little more complicated
00:50:20
and if you want to anything adaptive whereby midway through the trial you say gee i i think
00:50:25
i wanna add some more subjects and you know to appropriate statistical fix up for that
00:50:30
it can be a lot slower more complicated in a crossover design so they're
00:50:34
both they're both good they're both advantages but they both the disadvantages
00:50:41
c. g. i. e. four discuss all for all
00:50:49
we're we're look at what gets shot was actually i a shop
00:50:54
and group averages track how much information i help
00:50:59
yeah yeah i know it's personalisation it was for a slap
00:51:04
how how how are we going to address this issue i'll
00:51:09
doing l. l. l. k. as well a strike
00:51:13
well yeah ha ha ha ha ha it's i think it's sure it's
00:51:22
i think uh on the benefit side maybe slightly easier to address i think they're a lot of different ways to go about it
00:51:28
i think one way to go about it is is just to say you know what i'm
00:51:33
gonna pick the low hanging fruit first let me find the big group of facts
00:51:37
you know yours is in the future first let me figure out what works on average that i'll try to
00:51:41
figure out when you want to buy the that's one potential response i think the second potential response
00:51:47
is to say okay let's go when an estimate where there are big
00:51:53
big variability we take on faith that there is a lot of heterogeneity response
00:51:57
you hear this for example f. t. a. hearings all the time
00:52:00
pharmaceutical company comes i'll show some day on weight loss for example and you see a histogram
00:52:06
some people lost a little weight maybe even gain some weight some people lost a lot of weight so you see big variability and somebody says
00:52:13
can we have more information about those non responders look at that variability response
00:52:18
and we always wanna say it you have no evidence that this variable response all using is variability change
00:52:24
if responsible change we wouldn't need the control group the reason we have the
00:52:28
control group is that change can be other things other than response
00:52:32
the standard clinical trials standard harold booklet control is not at all
00:52:36
well suited nor is a standard to prick also design
00:52:40
estimating variability response gotta go much more complex
00:52:44
design like of multiple perry crossover or
00:52:48
ballot design what we published a paper in art class genetics covers alternating
00:52:54
you know within individual you could do that you could estimate to the
00:52:57
individual their fax calls to estimate the variability the facts or time
00:53:02
and then eventually you could try to figure out predictors of which treatment worked better for which kind persecuted those kind of things