Player is loading...
Clinical Natural Language Processing
Carsten Eickhoff, Assistant professor of medical and computer science at Brown University
Tuesday, Aug. 20, 2019 · 11:05 a.m. · 55m 09s
Clinical research has never been more active and diverse than it is at this moment. Research efforts span national and cultural borders and broad online dissemination of results makes insights available at a global scale with ever decreasing latency. In the face of these developments, individual researchers and practitioners are confronted with a seemingly intractable amount of material (approximately 1 Million scholarly articles are newly published in the life sciences each year). While highly trained human experts excel at making precision diagnoses, coverage, especially for uncommon conditions can be greatly improved. In this talk, we will discuss a range of (deep) machine learning techniques that provide automatic clinical decision support on the basis of large-scale data collections. I will present early and ongoing work on a) Predictive assistants in post-operative care of cardiac surgery patients, that serve as early warning systems in case of undesirable and dangerous complications. b) Automatic summarization of individually long patient records to obtain concise and topically targeted summaries for physicians. c) Data-driven diagnosis of rare diseases that individually occur too infrequently to allow clinical specialists to establish the necessary routine and experience.
Embed
Transcriptions
Note: this content has been automatically generated.
00:00:00
but um so exactly we'll be looking into ah a bunch of
00:00:05
projects that are going on have been going on for a while now
00:00:08
um some of those and i guess the things that i sort of direct across the atlantic and you are things that are happening around um
00:00:16
and we'll have i think throughout the talk is split between
00:00:21
are the concrete problem that sometimes the clinical domain or just or data set and somehow mandate
00:00:27
um and then i try to tie this back to white should we be maybe what care about no
00:00:32
p. in general not so much about clinical why should we care was to take away from the oldest is
00:00:37
concern me right to use 'em right when or if i go but i
00:00:43
believe a lot of problems general lies and we see them all again um
00:00:48
i see ah so this is the menu for today uh all keep it very short on the background
00:00:55
uh and then we'll look at two three problems so we'll look into
00:00:59
predicting adverse events identical environment so when things go wrong when can we
00:01:03
uh already get a look ahead on what to prepare for where things look like that might go wrong um
00:01:10
we'll uh look into i doing diagnostic decisions important especially for diseases that that's great
00:01:17
so where things are strange i think that look differently from the way
00:01:21
they normally do a strange um and co morbidity so multiple diseases together
00:01:27
ah ah strange right so so they present very differently from what the
00:01:30
physicians typically and that's what they see today that's what we try out here
00:01:35
um and then finally we're looking into text summarisation so how can i make long accounts for example
00:01:40
long electronic health records much shorter without losing any of the critical information that a physician should get
00:01:46
and how can it and and so so this is sort of the not so exciting part really exciting part i believe
00:01:52
is how can i personalised it's all good what is to stop to get the difference from but for the same age
00:01:57
um and and i'll uh hopefully tight is back to the journalists and what what
00:02:02
sort of the take away for people that don't necessarily want to work and it's
00:02:07
or um so i think that's probably small for most of you but um so i think this
00:02:13
is sort of where um the work that i've been doing this intersection is not really do but um
00:02:19
never mind um so i think most of our the work that
00:02:23
comes although i i somewhere in the wider data signs information retrieval world
00:02:29
uh in machine learning and and all sorts of sort of this is
00:02:33
greater clout and somewhere between the section of the union is i guess um
00:02:41
i don't know whether everyone knows what a chronic health records are what they look like or give many many overview
00:02:47
uh of what you can expect from those because they are a resource that we work with a um
00:02:54
so you can expect in your typical electronic health record and they're different from country to country for multiple possible
00:03:00
uh what you can expect typically demographics you know age gender or uh
00:03:05
who pays so this is the thing that you can typically expect so so you will know what type of insurance there half
00:03:11
um you will how vital signs of the in patients right so
00:03:15
you will know all um heart rates but pressures depending on how
00:03:19
intensively you'll be monitored it will be a more or fewer of these um
00:03:24
yeah if uh you're prescribed any medication we will no doubt um interesting we we won't know what you actually take
00:03:30
um so we just know what's been prescribed which is a big problem right so we have a lot of patients that
00:03:35
uh don't take commands because they already feeling better ah and sort of they they cancel on the therapies themselves
00:03:42
or um and this is especially a problem in parts of the us
00:03:45
uh people actually up on the way the mets right so if you on
00:03:49
painkillers uh and you want to make some extra bucks on the side
00:03:52
you sell this on the black market so which is again a problem um
00:03:56
lab results so if we order any lab test for you and it not working all of that
00:04:01
uh that will be in here the results um we may know
00:04:04
gino makes or star or makes um whatever we have about you
00:04:08
uh and then the the juicy part for us as an l. p. of
00:04:11
this you know those days of the unstructured nodes and this is something that
00:04:15
um standard data driven anything in madison often ignores right so we like to work with anything that's time series
00:04:21
because that's nice easy we know the range of values to come in so text is often completely ignored or
00:04:27
had simply poured into some form of topic model and then the sectors of being taken ah being worked with
00:04:32
um so and i think um we often try to
00:04:35
do something with this otherwise i somewhat what resource um now
00:04:41
the dark side of electronic of corrected is they were never ever intended to help any person
00:04:48
get better right so these things are primarily there for building we want to know what we did
00:04:53
i'm not in order to but do better but in order to write you
00:04:56
the correct bill and not perform any services that we don't get money for
00:05:00
so that's the sad truth um and that's really the primary purpose so that means that
00:05:04
uh a lot of their therapeutic too useful are kinds of bookkeeping only
00:05:09
uh being done in an afterthought on being done right so i c. d. coding for example
00:05:14
is a thing um so this this ontology all forty thousand i believe different codes that encode
00:05:20
um sometimes that you have that in court diagnoses that encode procedures that we perform
00:05:25
um and we assign all of these that apply to a patient to the record
00:05:30
and then the footnote to that is if we assign goals um that pay most
00:05:36
right so we know that if you have three diagnoses and one of them is
00:05:39
more expensive that's the one we'll caught the others we make or we may also market
00:05:43
right so there's a lot of these effects that you don't expect that first um that's that come out that
00:05:48
are being introduced by the fact that uh uh that this is the billing system not the heating system um
00:05:55
if we look at the the problem is so this is especially in
00:05:59
us electronic health care records to have a very brief polluted list summary off
00:06:04
patient has this this this and this um make sure patient takes this medication so the very the the absolute they just
00:06:10
a problem this uh at least thirty percent incorrect or outdated right so um there are people so we
00:06:15
found a single patients in hospital in road island to run with the cut to the finger for twenty years
00:06:22
right so just because that that fact that the cup notice laceration to the finger now healed
00:06:26
uh after uh probably two days after the visit was never be
00:06:29
updated right so this put patients just pleading for twenty yes um
00:06:34
these things grow relatively quickly so um if you're in hospital you'll oprah on average received
00:06:40
two new nodes per day so if you stay for months that's sixty issue notes um
00:06:46
and uh and approximately per one node per outpatient visit right so most
00:06:51
people in the us have about four and a half is it's a year
00:06:54
so every a you can assume forty five new nodes to come in for the average patient that's
00:06:59
not chronically all right so if you see doctors at a higher frequency this will grow much quicker
00:07:04
um i was seeing a bit um that i can become a big problem um interesting we
00:07:10
um so for each node eighteen percent of the text in there's new the rest is copy paste
00:07:15
um which again makes these things very very horrible to work with just because you have this does this
00:07:20
looks like an interchange right so you have all of this this stuff um repeated and repeated in every message
00:07:26
um and people right with elbows right so it's really a a vocabulary grammar
00:07:31
spelling everything is very nonstandard just because it happens pretty much on the side um
00:07:37
and that's sort of the the world that we're dealing with here so let's jump at
00:07:41
two offers problem um this is the most time series as um problem that will look
00:07:46
at the others are more natural language then take a adverse event production and this is
00:07:51
especially in the intensive care unit um these things look like this so this is um
00:07:56
very much what our patients look like here um and um so one interesting test is
00:08:03
common interests right so for me as a lay person who doesn't know anything about medicine
00:08:07
i go into these intensive care settings and i start counting
00:08:10
screens more screens you have the more scrupulous patient right just
00:08:14
because there are yeah no i mean that's the simple rule of thumb for for all of us that are not medical professionals
00:08:19
so two screens means you in the intensive care unit congratulations you could have seen that before um
00:08:24
if you have the the the more screens they roll next to your bed the bigger problem is right so um but i think
00:08:30
the the the interesting take away here is you see a bunch of these curves right c. d.'s all this time series that come in
00:08:36
um and these are all these concurrence signals that we measure about the patient rights when older
00:08:41
blood oxygen in the brain we know heart rates but pressure is all the good stuff um
00:08:46
i know the situation does that the doctors being faced with is take
00:08:50
all of the stuff that comes in and a minute by minute or sometime
00:08:53
second by second resolution and no reason about this patients was this patient doing
00:08:57
fine on that and how will they look like an half hour right um
00:09:02
so we in particular like you looked into um post
00:09:05
operative patients everyone had open heart surgery in this case um
00:09:09
and um and then there was in the first day or two that's a range of undesirable events that are
00:09:17
likely rare but frequently enough that we can sort of quantify them
00:09:20
here um so you may start bleeding again which now it's rather obvious
00:09:24
um dampen it's our form of bleeding that is especially nasty just
00:09:28
because you don't need to the outside but into um either your chest
00:09:31
cavity or into the little back that you're hard pressed and so that's that's more dangerous just because we might not see that quickly um
00:09:38
and then we have a rest and a bunch of other r. conditions um
00:09:43
and right now and this was a very interesting to me so the the baseline the state of the art is very much
00:09:49
a lot of training and a lot of experience right so
00:09:52
out with pleading so we started out looking at bleeding at first
00:09:55
so everyone after the uh the first surgery now this is idea if the patient leads this two ways of reacting to that
00:10:01
you give calculation factor so you stop the my um the the the bleeding hopefully um
00:10:07
or are you open the patient up again just because this is something more severe attend
00:10:11
traits and you want to really see second surgery what's going on and ideally you don't
00:10:15
give second surgery to people who don't need it because just that second surgery will also cause damage again
00:10:20
now um the rule of thumb is unless the patient is clearly spurting blood is that
00:10:26
the patient already lose a leader of blood it's all second surgery of not let's wait
00:10:31
right which to me sounded rather okay i guess uh as as a decision criterion but that's what med
00:10:37
school teachers you and that's what everyone does so the mission here is can we get better than that
00:10:44
um when we try to do this by a tracking all these curves that we have all the time and hopefully making the right decisions
00:10:51
um so what the doctors gave was was something like this so this was the markup off something that they would like to see so we
00:10:56
have our eight different complications here so for example be doing what it
00:11:00
sees tampa nights or whatnot and you want to see a different time horizons
00:11:05
uh what's the profile of these also how likely are these complications and you may want to
00:11:11
um select single ones of those in show how this is behaving over time rather
00:11:15
than checking seventy ravel use all the time and making your own reasoning of this
00:11:19
i just want to see a single probability of you developing a
00:11:22
complication within time from x. that's that's the the mission statement yet
00:11:26
um does work that was done in but then we had thirty thousand patients
00:11:31
um we looked at them for twenty four hours after their initial surgery um and we
00:11:36
tracked fifty a different markers here uh these
00:11:40
aesthetic things such as age gender demographics um
00:11:44
what type of surgery you had originally uh we have their vital signs which
00:11:48
are pretty much live um we have the labs to come that come in
00:11:52
add a hour by hour sometimes to allah sometimes not at all anymore
00:11:56
right so whenever a doctor goals or the something it will take an
00:11:59
hour to for the results to return and then suddenly we have one measurement point of well here's your red let's all control things like that
00:12:06
um and then we have the the free text notes that um how both uh describing this
00:12:10
actually but then also the the camp process of the nurses will document whatever they find their um
00:12:16
and we have all of these um adverse events that we catch predictive
00:12:20
labelled explicitly so we know when and if they actually happened for the patient
00:12:24
um and i would try to turn this into production problem not the architecture that we have
00:12:29
here is relatively straightforward um so we start out with our wouldn't readings on the on structured notes
00:12:36
um we have a c. n. n. not goals over a a bunch of these words
00:12:41
to uh give us a sentence representation and we do the same thing in another layer
00:12:45
where we go over the sentences in the node and eventually uh reach
00:12:49
um high notes representation items on each node we start to reasons every
00:12:54
time a new textual note comes in we update all believe on the
00:12:58
patient state um and and yet that's this level we have these easy um
00:13:04
and z. o. all the uh the labs the vital signs all the structured information
00:13:08
if we also may know about the patient right so that we just concatenate here
00:13:12
um with this uh unstructured representation and this will be reasonable so no
00:13:17
initially we started out just making predictions over here right so it's
00:13:20
all likelihood of all come given this feature representation of this node
00:13:24
um that works reasonably well and we have all last year for that um no
00:13:29
interesting we um what really helped us um get more performance out of this was
00:13:34
to compose this original loss here um together with the
00:13:38
second component what we say every node here not just
00:13:41
eventually over time all the notes that i've seen but every single node here should be a good predictor
00:13:48
of the overall outcomes ideally i want representations of what this does is it forces
00:13:52
or representations early on already so from the anemone says or from the subsequent notes yet
00:13:58
two already also be good predictors rather than this overall thing here being productive which which really
00:14:03
uh first of all hopeless converge much more quickly um and second of all gave us i'm a
00:14:09
i'm a much better overall performance which was really interesting to see that this this kind of target application
00:14:15
i'm sort of this in the same spirit estes does joint training objectives that people look at
00:14:19
at the moment right um this this really change the game he and helped us a lot huh
00:14:26
so if we look at the numbers again i'm pretty small but let me quickly describe to you
00:14:31
um so we have for bleeding mortality indignant kidney failure i'm always done the numbers here
00:14:38
and um what you'll see is the real always a significantly better
00:14:42
than the space lines okay that's maybe that's exciting and something like this
00:14:46
i'm here we draw this out all the time so blue is the um
00:14:51
is the networks that uh that we built he actually this very anti r. doesn't of conclusions anymore but but the
00:14:56
recurrent neural net but the falls very much does does today
00:14:58
or approach of goat from sentence representation to notre presentation um
00:15:04
and here and right you see uh the textbook baseline so the algorithm that the task textbook says you should
00:15:09
follow right so this is this is a little plot out of the patient or not right and all the
00:15:15
i'm here what we see is is quite interesting this this
00:15:18
time parsed since the original surgery right and we see that
00:15:23
most of these methods learned all the time so in the beginning the patient is just doing badly you just
00:15:27
came out of a long surgery i'll of course your system is in all kinds of states of this right
00:15:31
uh and then as time passes we see that these two groups of patients
00:15:35
the ones that will do fine and the ones that really need second surgery
00:15:39
they will slowly start diverging rights only becomes easier and easier telling them apart
00:15:43
and we see this general pattern i mean there's there's some fluctuation here and there
00:15:47
what we see this general pattern for most of them that we
00:15:49
start all out um i'm the the textbook algorithm even not even arbitrarily
00:15:54
um but all of them uh tend to get better but with the release all that margin between those um so i think what i
00:16:03
find really exciting about this uh which is more or less a
00:16:06
bread and butter machine learning problem and is the fact that we actually
00:16:10
went now to the next step and prospectively deployed this so this was all retrospective for it so we had all the stale data
00:16:16
now we're at the point where we have these machines really deployed by the bedside
00:16:20
um and physicians can can actually see all the likelihood of the of a bleeding over the next five hours
00:16:26
is uh is is twenty percent for this patient worse is much higher for that other one
00:16:30
and this is really really interesting because now we're getting to the point we
00:16:34
can actually um proactively uh try and do something about these things because often
00:16:40
um how our colleagues in the in the intensive care unit they say oh
00:16:44
um these patients all look fine you have twenty of them and you alone overnight
00:16:49
um so you you spend most of your not drinking coffee and suddenly you have three people that need to be the animated right now
00:16:54
right so this is sort of what you want to avoid so you want to really make sure that if someone is likely to need a second surgery
00:17:00
which actually have an o. or perhaps that you have a team there and the surgeon who's hopefully in some state of awaken us
00:17:05
right so um which is a bit of a of a resource planning problem right so
00:17:10
um but they um they are right now using this ah and it seems to to be
00:17:14
working very well for them even to the point where we have a bunch of people that
00:17:18
we are quite confident would've made it without this thing which is uh i think to assess
00:17:23
non medical professionals helping save someone's life is something that we don't really do all that often so this
00:17:28
is really cool and i think unexciting problem that goes a bit beyond just demonstrating all we had this good
00:17:33
add forecasting does we label that doesn't really mean anything right
00:17:36
so but to really carry this over if does being good at
00:17:40
this task translate into lower mortality better patient outcomes i think that's
00:17:44
really the test of time so we can't really qualify that yet
00:17:47
um but hopefully um i would guess and a half euro so we would have seen enough
00:17:51
episodes so that we can compare standard of care with this with this new model here um
00:17:58
alright so um something very different um
00:18:04
diagnostics or especially rare disease diagnostics um i'll start with the case that we so
00:18:10
this is a work that was done in zurich at the uh university hospital there
00:18:14
and um so they had a case is an actual patient
00:18:18
yeah and the started out someone presented with stomach ache and constipation
00:18:22
not all that uncommon um but they had that for a bunch of weeks so
00:18:27
this person uh eventually went uh to the emergency room uh and uh presented with one sentence so
00:18:34
all kinds of checks were being run an inconclusive the prescribed some light medications and the person how
00:18:40
um some weeks go by sometimes the patient feels better sometimes the patient feels worse
00:18:45
um but by and large nothing goes away actually the the symptoms become stronger so we run some extra blood work
00:18:52
so we find an email so the blood cells of the tail and a bit when key and they do all kinds of funky things
00:18:57
but it's unclear why so all the usual courses for any uh are not given so we again
00:19:02
person prescribe some medication sent to spy person home
00:19:05
i'm at a a one six everything becomes worse um
00:19:10
so we checked for and you're is um in in a lot of blood vessels confined any of
00:19:15
those so we run a lot of tests send the person home with some medication so in between here
00:19:20
um the entire system of this patient start shutting down right so this person
00:19:24
who's i say the kidneys shut down so the entire system just goes into standby
00:19:29
um and and eventually and this took almost two years
00:19:32
of this person uh incurring serious and also a permanent damage
00:19:37
uh someone runs a lot of lot panel so something that you wouldn't do here so
00:19:42
uh and test will lead in the in the plot and then they found that this
00:19:45
person was twenty five times over the absolute critical threshold value off do not go there um
00:19:51
which is interesting because nobody would check for this because this person had had an office job right so there was no all
00:19:56
uh in switzerland in an office environment is to expose or too
00:20:00
heavy metals absolutely not a given right so super rec yeah um
00:20:04
and then eventually they could be treated for and that's all fine but what the doctors found out then
00:20:09
uh in the end was that this person had been importing herbal remedies from abroad so this person had imported
00:20:15
herbal medicine from indian or your basic stuff uh and this stuff was changed in that
00:20:20
because they had taken this for eight years already and the symptoms that only started now
00:20:25
um nobody made this connection but was always heavy metal poisoning so it takes a
00:20:29
while until you reach this critical concentration in your blood and suddenly you system starts offering
00:20:34
right so nobody made this connection this person the patient themselves didn't even mention that all at
00:20:40
using these we're t. slide so but they're all not prescription a
00:20:44
a great my taste medications so uh this is one of these cases
00:20:48
that um didn't go so well let's say right so this this is
00:20:52
one of the strange cases um when nobody use all the right thing
00:20:57
even though all of these hear us in terms of lead poisoning
00:21:01
just that poisoning is so rare that you wouldn't really assume that
00:21:04
right so and it's it's really if you see your position in the flu season it it hardly matters still hard look at you you'll have the flu
00:21:10
just because that's what they see day in day out right so that's that's the very rapidly so
00:21:16
no this wasn't really a disease as such because it's a poisoning but it has all the the the
00:21:22
telltale marks i guess of a of a rare disease as in it's hard
00:21:26
to spot just because it's infrequent that and therefore is this now truly rare diseases
00:21:32
um typically have some formal threshold so um i think
00:21:36
this should be the u. s. definition of a in a
00:21:40
population of fifteen hundred people no more than one half this
00:21:43
individual disease right and then you have all kinds of ah
00:21:48
so normal diseases to uh the rare and alter read diseases um
00:21:54
ah but interestingly there are many of those so we're looking at seventy eight thousand for diseases
00:22:00
and if you sum up this volume you get to something like ten percent off
00:22:05
i mean this is the us number the european numbers very some not
00:22:08
a similar right so around ten percent of the first world population have
00:22:12
a rare disease i even though individually these things are hard to spot and uh
00:22:17
and a tricky and these are rare things such as uh see like right so gluten intolerance the actual one not the
00:22:23
imagine what so people who truly suffer from from this this is often spotted very late it can cause
00:22:30
all kinds of of release of your cancer is it can uh make your bones go to mash right so
00:22:36
if you truly have celia like that's that's a really tough thing and for a long time it just looks like a very irritable stomach
00:22:42
right or ball system so there are there are a lot of these things
00:22:45
that we believe are pretty every day everyone knows someone that can't deal with student
00:22:50
um right but it's it's sort of these this is sort of
00:22:53
the level of things that we call wrestled very much nonstandard um
00:22:57
so now here um we have a this problem off lack of training data and this applies to
00:23:04
both the human physician as well as to standard machine learning right so if you think of this as a machine learning problem what you
00:23:10
would do is you would say all this one class per disease that's
00:23:13
easy that's forty thousand does approximately the number of diseases in the world
00:23:16
um and all the patients that i hate to observe with this disease go
00:23:20
into these classes on on my uh my class conditional model and i'm absolutely screwed
00:23:26
right because forty thousand diseases with a highly a skewed distribution means that in for most diseases your
00:23:31
zero training data or very little right um so there's this i'm not what we want to do
00:23:37
and then also um the same holds for the doctors who may have heard about many of these conditions in med school
00:23:43
but we don't in their daily pack to see these things right so um your
00:23:47
uh your way of treating a patient will always be a or comes raise uh what's the easiest
00:23:53
and best explanation well that's typically the very common stuff and that means that if you truly do have
00:23:58
um how rare a stranger condition i can often take you
00:24:02
months or years to find this thing so forcefully at for example
00:24:05
the number is close to two years on average that it takes people to be diagnosed with this after this into the show
00:24:11
right so you have some some severe enough that you go see a doctor and not just tough it out because say
00:24:16
but of the area um and uh and from then on we are talking ah i believe it's twenty
00:24:22
months uh so until c. d. x. actually recognised on average right and that's something that's already reasonably frequent um
00:24:30
if we look at the damage that's being caused by there is it's um
00:24:34
we're looking at around a hundred thousand us dollars
00:24:39
for patient and here in just treatment costs right so this is just treatment that goes nowhere
00:24:44
because we treat the wrong thing that we believe the patient may have um and of course um
00:24:50
if we waste time treating the want wrong thing you might actually get worse
00:24:53
and worse uh and especially for very young patients the country articulate what's going on
00:24:58
uh there's a big problems are rare diseases are involved in about a third of all first year infant mortality cases
00:25:04
um and if we generalised is to just uh and this diagnosis in general if you're really looking at
00:25:10
um tens of billions of us dollars just uh just in the in
00:25:14
the in the united states i'm gonna be in cost them that much now
00:25:19
because
00:25:20
we all thought that and there's still a bunch of studies that show that released on that machine learning will not help you
00:25:27
here right so virtually any traditional machine learning paper that you see
00:25:31
that does anything in the in the range of diagnostics will pick
00:25:35
between one and twenty diseases and we'll show you that they can
00:25:38
recognise this stuff with money for us that ninety five percent accuracy
00:25:42
that's fun but these are all of the things that the physicians already sports perfectly
00:25:46
right that's really it ahead of the distribution this is the stuff where you poor condition have tens of thousands of training data points
00:25:53
so nobody ever looks at even a hundred diseases let alone a thousand
00:25:56
body forty thousand that we actually have the real world so where you
00:26:00
would actually help the physician is where all machines really struggle so this
00:26:03
is why we turned away from from the typical um machine learning here
00:26:09
and uh and try to do this via information retrieval also try to phrase the whole thing
00:26:14
as a search problem rather than the classification problem um so uh all quickly sketch what that look
00:26:20
so we start with our first time point here so someone comes with
00:26:24
a stomach ache constipation and probably the doctor will write a bit more than
00:26:28
just this um and what we do is we use this as a query right so we take your electronic health care record at this point here
00:26:34
um we use this as a query so you need to do some filtering named entity recognition and all that good stuff
00:26:40
um but let's assume we have a good way of mining queries um from from such electronic have corrected
00:26:45
um what we do is we rank biomedical papers right so case reports study is
00:26:50
um what people describe historic patients not just in your own hospital
00:26:55
but just anywhere so so close globally published um we get this this kind of ranking
00:27:00
and um and then from newspapers you we can mine ah
00:27:03
again using information extraction things like that we can minor ranking
00:27:08
over our um over diagnoses right so we could try using just this one data point here
00:27:14
and say oh i'm stomach ache and constipation
00:27:17
they are um caused by menstrual um um
00:27:22
of menstrual pain as we had one right but um it might be that your
00:27:26
heart irritable bowel syndrome that's i read so that will be something that's right highly
00:27:29
and some weight on this list i bring two hundred or so we'll see that point something
00:27:33
because that can cause that's okay that's not helpful because nobody will scan discussed so far down
00:27:39
we also have is we know that at the next point in time someone runs a bunch of different tests gets a bunch
00:27:44
of different results here and again we can play the same game right so if you have on this form of any any idea
00:27:49
we uh can rank this and we can see that any me as course by and efficiency is uh and
00:27:55
maybe the vitamin b. deficiencies and so on so their courses what isn't somewhere during eighty
00:28:00
um will again see lead poisoning again that's not helpful because no physician has the time to read that stuff for
00:28:06
and all the interesting thing that we can do is we can fuse these results all the time so
00:28:11
the fact that um because for this and the cause
00:28:14
without and this and so on right are largely unrelated
00:28:20
but the true cause for all of these things see uh should be consistently ranked among the
00:28:25
hopefully more um was the top it's of these uh of this list that we produce yep
00:28:29
so basically if we start joining these rings and having metrics
00:28:33
such as off just sum up the rank and which you appear
00:28:37
here right so you eventually get to a ranking that hopefully
00:28:39
washes all the true diagnosis the consistent because all of the sentence
00:28:44
um whereas um all the the other stuff you uh so also to see me uh uh and whatnot
00:28:51
uh i'll be in our being punished and being being put on and that stride if
00:28:56
and this is something that interesting the works extremely well so um we uh run a
00:29:01
bunch of uh experiments here so we talk a a million and a half each puppet papers
00:29:06
we uh i took twenty two case reports off particularly hard problem so these are not the people who
00:29:12
in flu season come with actually if you end up being diagnosed with the flu or not being sent home
00:29:16
so these are uh i had big cases where our uh our collaborators at uh the only spit out indirect
00:29:22
said oh this is hard these were apart knots that we that we root around with for a bunch
00:29:27
of mons and these are tough to cage uh and ends typically it's it's because the condition is rare
00:29:32
um because the great patient is a bit strange all because they have multiple things and
00:29:36
you often have combinations of multiple diseases that just to get a look like nothing else um
00:29:41
so all i do studies twenty two cases that we started out with um
00:29:45
yeah a different diseases in here so some of them are actual multiple times
00:29:49
um and we have three two nine episodes of electronic health care records steps in time
00:29:54
um for each of these ah i'm ah i'm weak compared this this this ranking quality now uh our
00:30:00
baseline is just a patient text classifier right so if you work to approach this in a very standard
00:30:06
um machine learning way you would say okay that's one model per disease um and now you look
00:30:11
at the posterior probability of having generated out patient on the the model of that it's it's right um
00:30:17
and you'll just count as as a um as an and one model
00:30:21
of right so and the thing with the uh with the highest probability
00:30:24
of having generated that patient will be ranked highest and so on and
00:30:27
instead of just doing classification we really rank so both methods give us ranking
00:30:31
so he wrecked by the posterior uh and the other case we rank by um what i'll retrieval model gives us um
00:30:39
and then everything that mentions a particular disease goes into this thing
00:30:43
now um does one interesting parameter and this and this is this is what we call a capsule be this is the kind of what
00:30:50
do we consider a diagnosis right so how many diseases are you willing to uh
00:30:54
to uh to diagnose and we'll see on the next slide ah how this behaves um
00:31:01
so he uh our performance metrics m. r. r. which is basically one
00:31:04
over the ragged which you recognise the correct diagnosis so if you have
00:31:07
a trend one it's one and otherwise it goes to point five one
00:31:11
third and so on right so higher is better expounded in zero one um
00:31:17
now if we half hour triple approach here we get to score off close to point five so
00:31:22
meaning you are on average joe correct diagnosis will be in the top two on result trying to
00:31:27
um which is useful right so this is not a drink eighty anymore nobody needs a scroll you look at this list
00:31:33
and you test for a bunch of things and on average you will have found um what's going on here
00:31:37
now let's look at the next page in a method here and here in brackets i show you how many
00:31:42
uh most frequent diseases uh we're looking at right so the things which we have most data
00:31:47
and we always make sure that the eight diseases that are truly the ground
00:31:50
truth i artificially injected you're right so here we have twelve run those plus
00:31:54
i'm helpless the eight diseases that actually the target which we sort of help this model to cheat a
00:31:59
bit um and here we're looking at as we make
00:32:03
this problem more realistic what happens you're really right um
00:32:08
so if we're willing to recognise a hundred or even two hundred fifty diseases uh something like this
00:32:13
um which is sort of we're all machine only papers and um then we get to know and the and the
00:32:19
top four five francs of the result list you still see
00:32:22
your diagnosis but these all not the the the rest stuff
00:32:25
so had we not injected artificially these super rare conditions in here in these and and these groups you
00:32:32
uh these things would never found that if you make the problem more realistic if you're willing to say
00:32:36
okay out of forty thousand diseases that they're on the world we're willing to consider the top one thousand
00:32:42
then already you're looking at a result list of light and fifty right and that's already arguable whether that's
00:32:47
still useful for position test for fifty different diseases really
00:32:51
do the deaf or differential diagnosis for fifty different diseases
00:32:54
that's something where way we're not entirely sure whether physicians will actually have
00:32:58
the time and the patience just all right and and that you would have
00:33:01
to look at forty k. actually in order to properly this whereas basting here
00:33:05
just does matching right so we have in this retrieval approach we never have
00:33:10
class conditional models so we never go up and say oh congestive heart failure has this following model we
00:33:15
only do matching rights if you your description is similar to pass patients descriptions we links you to each other
00:33:21
right so which i believe he uh gives it was really an edge on these things and it would also mean that
00:33:27
as soon as a new diseases recognise we're and principal able to um um to recognise
00:33:33
it as soon as as being published on right where is here you would have to
00:33:36
update your model and it would have been ah i class conditional model with the single
00:33:40
training data point which again is probably not the most abbas thing in the world um
00:33:46
and here we played around a bit with what happens if you change the collection size that you're half
00:33:53
um so this uh this number of million articles that we uh uh that we're using in a belief
00:33:58
we have a hundred random stations of drawing these um except for you because is or anti data set
00:34:05
and um and we can see that there's does does quasi linear a relationship between
00:34:10
more articles mean you have a better chance of having observed the patient that behave
00:34:15
similarly we ugly to your stride so the more evidence you have
00:34:18
the the great it's your chance of actually catching um the true diagnosis
00:34:24
right um so this was really interesting and this is something that we are um
00:34:29
still doing together with ins of the talent bar and and with the university hospital in zurich and a bunch of others
00:34:34
um where we're trying to bring this not now really out to the patient right so which again
00:34:39
this this step uh experiments done paper written a a great paper being side it that's all fun and games
00:34:45
but i think that would the real differences being made once you once you try and bring it to
00:34:49
the patient which is hard because that means you have to build actual systems that are robust enough to
00:34:54
take millions of requests ideally and and whatnot so this is sort of the challenge that we're facing up and this um
00:35:03
okay so the final and this is um very early stage
00:35:06
ongoing work so this is not yet published but i'm i'm super
00:35:09
excited about this and that's why i wanted to tell you a bit about it is uh this idea of personalised text summarisation
00:35:16
so i'll very quickly tied back to early so uh u. h. r.
00:35:20
s. uh grow fast right so two notes per day one up a visit
00:35:24
depending on what you have if you elderly or chronically ill you'll see your physicians
00:35:28
a lot more often you have accumulated a lot more most often some more big stuff
00:35:33
um uh and therefore you will just have ah
00:35:36
more that bills that i need attention right so especially
00:35:40
for ah loyal customers as we call them right so people that don't move around much so you
00:35:45
live uh and and see the same g. p. for thirty years they have a huge record a new
00:35:50
uh if you're very all your record also grows very quickly right so so these are very long documents now
00:35:57
interesting lee and these numbers are from the us i don't exactly know how
00:36:00
that translates into your but i would assume it's not much better here um
00:36:05
so your physician will spend about twice as much time
00:36:09
with documenting what they see on you then they spend see
00:36:14
which isn't really great right so for every five minutes uh actually doing
00:36:19
stuff with you they take ten minutes writing on what they saw um
00:36:24
so a lot of this is not being budgeted for in the time
00:36:28
allotment that a physician has with you so that means a lot of that
00:36:31
happens i'm at home or maybe not well see um so this whole
00:36:37
record keeping is is very problematic at the moment so a lot of people
00:36:41
uh work with the use a text synthesisers also speech to
00:36:45
text kind of engines to try and get this this uh
00:36:48
this uh scribe process out of the way but still at
00:36:51
the moment um a lot of physicians especially primary care physicians are
00:36:55
i'm very frustrated but it's process not translates potentially into medical error rights if
00:37:00
this constantly is something that you do in a bunch of hours at night
00:37:04
maybe you're tired or maybe or frustrated enough to ah to overlook things uh
00:37:08
i similarly um it's pretty proven that this leads to burn out among physicians
00:37:12
especially primary care physicians ever pretty high rate of just going out and doing
00:37:17
other stuff uh i'm going in for a nice cushy from a job instead um
00:37:22
okay so i'll we're going to try here is can we use automatic
00:37:26
summarisation for this right so the idea being you have this long health record
00:37:30
you want to uh give the position an automatic summary that's
00:37:33
hopefully up to date as all the important information doesn't miss anything
00:37:37
ah and it's ideally much much shorter right so binds like less of that time um in preparing for the next
00:37:43
is it's a reading up what's this patient again what happened to them since i last saw them and so on um
00:37:51
very quick recap on on the market magic summarisation comes into fundamental flavours
00:37:55
um the old fashioned one is extract of summarisation uh that's very much like writing
00:38:00
that run some letters back then right so you look at uh all your original material
00:38:05
uh and because generating language uh prior to twenty thirteen was really hard so what
00:38:10
we do instead is we look for language that humans wrote so that's probably grammatical
00:38:14
um which will take phrases or sentences out of theirs and paste this into our
00:38:19
summary right so um and that's really just you it's very much information retrieval based
00:38:24
so you have some form of query ah and you go for all
00:38:28
the juicy facts uh about the patient and you place them in here
00:38:31
and since that human wrote that for you you never really have to
00:38:34
start generate having the machine to learn how to rights as human language
00:38:39
um and then the uh the the somewhat more popular approach at the moment is abstract of summarisation where the ideas much
00:38:44
like a person would reach the text or the event or whatever you have so you read all the available information and
00:38:50
then you go way you think about this and you're right i hopefully well formed summary that and someone reads right so
00:38:56
you really get into the business of generating language so this is um a framework that is from twenty fifteen i believe um
00:39:04
and what this does is you have your um your original sentence here so this is the long text
00:39:09
uh that's the summary that you're generating one word at the time um q. encode these uh these
00:39:15
words yes we have typically some form of of recurrent neural network i given attention distribution across it
00:39:21
that's being informed by what have we already generated right so i know
00:39:25
what i've already written about and where i am now in the generation process
00:39:28
um i know this gives me this this context vector here from which i then
00:39:33
sample um he just does what distribution right end up with uh tell me that
00:39:37
of the sentence start in germany i would want to say that they teach someone and then i'll see that
00:39:42
there is such a argentina in the two zero when and whatever going right
00:39:46
so very much this is the the basic architecture notices a bunch of problems
00:39:50
um so the next iterative step was um to do this pointer generator
00:39:55
um and this is the ability to basically say you sometimes want to copy
00:40:00
you largely want to generate text but once in a while you want to copy text and is mainly
00:40:03
due to all the vocabulary problems so this guy he requires that they be word embedding fall token see uh
00:40:11
well who works in neural that much language processing knows that we always have
00:40:15
to constrain our vocabulary because otherwise it doesn't fit into our jeep use memories so
00:40:20
the vocabulary sizes the first thing to to die right so
00:40:23
things like two zero is likely to not have a word embedding
00:40:28
alright so um and we would be able to generate this token here right so this architecture here no gives
00:40:33
us the ability to say oh yeah i want to uh generate as we did start germany beat whatever right
00:40:40
um i have all of this but then we have this p. jen here in dispute genesis which if that
00:40:44
says okay there's some probability with which you want to
00:40:48
um indeed sample from this distribution here generate any word
00:40:51
and then there's the the complement of that which says maybe you just want to copy a token for me
00:40:58
right and this is the nice thing because that's as so even though this thing here
00:41:02
will end up being some unknown word right so that i don't have a proper embedding for
00:41:07
i may be able to say off actually now the best option here to take a is
00:41:12
ah maybe a a sample or copied is on all worked right so i'm just just cost me
00:41:18
um and is ah has really revolutionise both machine translation and
00:41:23
uh and summarisation which are very similar so whether you translate
00:41:27
from english to georgia english or from english to friends uh i'm french turns out to be um
00:41:34
a very similar tasks also i think since all these these neural architectures out there
00:41:39
a machine translation and summarisation work much in the same way um which is quite interesting um
00:41:47
one big problem that many of these architectures how far repetitions
00:41:51
so many of these models lose themselves in repetitive uh repetitive clauses
00:41:56
and then and then and then and then and hopefully that's not well the third eventually but it really doesn't
00:42:01
uh isn't the best use of the of your vocabulary and of your of your space that you have here
00:42:06
so a lot of the sequence to sequence models have have that kind of problem
00:42:10
um coverage is a different thing that was um proposed for some of these models
00:42:15
which basically says in your attention layer so in in this guy here
00:42:18
right um you want to make sure that every german every phrase get some
00:42:24
uh accounted for amount of attention right so if you've already i tended
00:42:28
to this word are not right which is this some here over this twenty
00:42:33
i'm i'm probably i don't want to draw from that again and generate that again right so and that basically
00:42:38
uh we can use that even the attention they answer all because in the last few turns i've already sample from this
00:42:43
word and i don't want to do that again um or i can do this in the last function what you say
00:42:49
overall i would discourage summaries that are highly repetitive just because
00:42:54
they pay too much attention to the same tokens all the time
00:42:57
i i'm not actually lose too much more natural appearing um summaries and we'll see
00:43:02
in a bit that we're not quite there yet um but that is a really helps
00:43:08
okay um i'll gloss over this in a bit so the idea is that
00:43:13
physicians are different and they want to know different things about the patient right so
00:43:16
think about your cardiologist and your eye doctor they probably have very different questions to
00:43:20
watch you uh and you the idea on the page for this whole thing is
00:43:24
um why don't we summarise the personalised manner um unfortunately that's not something we
00:43:29
can do with state of the art that's right so they say fuses source
00:43:34
generate something and if that overlaps highly with uh with the target well that's good right um
00:43:40
so what we try to play around with our our topic models
00:43:43
right so we try to estimate topic distributions for these original text
00:43:47
and then to use this distribution in making the summary right
00:43:51
and now i can fiddle with it so i can say oh
00:43:54
actually painless this topic because i don't care so much about this or um altogether remove that
00:44:00
and if you think about the general news domain right so if you think about your news analyst
00:44:04
they want to read up about certain developments and they might be the expert for the asian market and they don't really care about developments
00:44:10
in europe on this it has immediate consequences on ages so summarise this
00:44:14
thing but really penal lies to europe angle on this and focus on
00:44:19
right and we are able to do this and also now we have a vehicle that says i'll have all these these little levers and i can say or
00:44:25
give me more that less of that while still making about that summary of this so this is the the basic idea um
00:44:31
this again is small uh and all just very quickly give you this idea
00:44:35
off now we have this topic distribution right so modest attention all the terms
00:44:39
um but also this original topic distribution are we can now enforce that is summary
00:44:46
match either the identical topic distributions so make sure you are very similar
00:44:51
in distribution of attention that you give to certain topics not there's individual words
00:44:55
or i can as i said fiddle with this enforce certain different distribution on this whole thing right and it's basically
00:45:01
goes into account expect tricky of um and this is then the thing that we either sample from what we copy from its original thing so
00:45:07
very much same basic idea just this topic component gives us a slightly
00:45:12
obstruct few on on the on the language that's being used to um
00:45:18
so the h. o. stuff is still in the running but we do have results on new summarisation sold somewhat
00:45:23
different domain exactly same architecture so i'll show you a bit of this and has some promising early results um
00:45:30
so this is the typical if you do summarisation this is the data set that most people use so this is news stories from c. n. n. the daily mail
00:45:37
um we have somewhat shy of three hundred thousand articles and summaries for them
00:45:42
um articles are typically around eight hundred words long summaries
00:45:47
of the reference summaries around sixty so they're pretty short
00:45:51
and um this is our nice little due diligence comparisons of these two here out
00:45:56
what we propose and they're doing pretty fine in comparison with uh with the baselines now
00:46:03
let's um look into one big problem again repetition right so we do news coverage and
00:46:11
still we saw that all this to still all in in all these baselines with coverage mechanisms
00:46:16
uh there's still this problem of them repeating themselves so uh we tracked is now so what
00:46:21
you see here is the percentage of enron so you're the grounds bigram trigrams and so on
00:46:26
being repeated using the vanilla point to generate a blue
00:46:30
pointed generator with coverage which is sort of the baseline
00:46:33
uh our method negatively and then in red uh all method here with this coverage and
00:46:38
if we look at you know grounds there's not really much of a difference just because sometimes these words you
00:46:42
need to you reuse right so the this these are really atoms of the message that you want to send
00:46:46
i you'll have to use them now but already as we look at bigram see you can
00:46:50
see that as we use um the coverage mechanism
00:46:54
on top of topics now we can really avoid
00:46:57
a repeating all cells so much just because we don't know the topics gives us this vehicle that say oh
00:47:03
yeah you you do want to pay attention to this topic here
00:47:06
but since you've used those identical words before out make sure you use
00:47:10
synonyms all the things right so and this is really interesting i'm
00:47:13
here in this in this uh uh context i guess what you get
00:47:17
much more natural appearing so if you look at the extent of these bars
00:47:20
so twenty not fifteen percent of all six problems with a pretty long sequences
00:47:25
uh being repeated but virtually all of these methods here how where versus this guy was probably at that
00:47:30
to what three percent party of um which is much much nicer in terms of the summaries that you get
00:47:36
okay so i promised you personalisation but didn't do personalisation so far right so i only said
00:47:42
the distribution of topics that you see in the source um and now we played around with this as well so our first example
00:47:48
here and i'll send you the slide so that you can actually read this uh at home because there's maybe but a small um
00:47:54
but what we did here was um this is the reference summary um this is often
00:48:00
the uh summary offered and now what happens if we take out anything related to health care
00:48:06
out of the story right so um we have a person with a nice murdering a
00:48:10
patient in a in a hospital um and then and then there's a poisoning k. is right
00:48:16
um and of vanilla summary here we'll talk about hospitals
00:48:20
entails a and nurses and whatsoever and here we still correctly
00:48:25
describe out what's going on but we don't use these words
00:48:28
sorry right so we talk about a person murdering uh that that's a poisoning other people
00:48:33
um but without really describing the hospital angle to this right so and these are really
00:48:38
very early results of this actually working out quite nicely rights and you can imagine that there are some use cases for this
00:48:44
um in the age all the main uh we're not quite there yet just because summarisation is at the top for their um
00:48:50
but um so that's sort of what we're doing okay let me quickly in the interest of time also do a
00:48:56
very quick ah gloss over or why should we care if
00:49:00
we don't work clinically white was this even interesting or um
00:49:05
representation burning i believe um as a a hot topic
00:49:09
right now and everyone's doing it in some form of that
00:49:12
um and that's pretty easy right i mean just pouring anything into birds and believing that that's the new
00:49:18
uh new truth right or excel net or whatever your favourite term transformers at the moment ah that's pretty easy
00:49:24
but ah learning meaning for representations as much tougher right so
00:49:28
do these things really represent anything that corresponds with human intuition
00:49:32
right um this is broken latex formatting but uh so think of explained ability
00:49:39
right so if i go into the whole business of explaining why a i
00:49:44
predict a certain class label right then it would be really nice if the atoms
00:49:48
of the message on which i'm predicting so the word and readings all the way
00:49:51
to begin if they do mean anything to people right so if they're really um
00:49:56
choose we had a a co occurrence um generated vectors that's difficult um
00:50:03
and uh and we saw this for example in the adverse event production case right where um
00:50:09
tools such as last functions that force representations to be expressive already right so not
00:50:14
just tools but eventually in some really complicated machinery of being used but really forcing
00:50:20
the the meaningful miss all the way into the the and on all the bearings for example is useful
00:50:25
a lot of people use these joint training objectives where you could train on for five tossed right
00:50:30
to just make sure that you don't over fit too much to your problem at hand and
00:50:34
often you get a lot of good results from that and same for transfer learning so not just
00:50:39
uh if you don't have enough data but also really if you want to make sure that what you learned there
00:50:44
corresponds somewhat to to a general meaning uh i think is very important
00:50:49
um hi all the problems a rare events that's
00:50:52
that's probably my my favourite a problem at the moment
00:50:57
so i think machine learning is really great opposites catch right so ten thousand images of each
00:51:01
clause and you'll get fantastically autistic be good at this um but does that really mean anything so
00:51:07
what happens if the number of classes and the number of data points per
00:51:11
clause uh if not grows or excuse right so and that's that's a big problem
00:51:16
so there's a lot of really really exciting stuff happening in the
00:51:19
whole one shot zero shot learning community right so what if i
00:51:23
if you only a single training example polka so maybe you've never seen an example right so i only tell you this is
00:51:29
all the class now predict that thing but you have don't know training data um so there's super exciting stuff here and i
00:51:35
um so we saw a bit of that on the whole rare disease diagnostics um example
00:51:41
and i firmly believe that uh things that work in this retrieval style and if you look at most of the
00:51:46
of the more than zero should learning algorithms to all retrieval and so on some point rather right because you always say
00:51:52
all project these things into space and then look for neighbours or the uh look for gradients that you can actually generalised between them
00:51:58
so so very interesting that we actually build a search engine here uh in the
00:52:02
middle of all machine running approach with this um and then finally uh i think here
00:52:08
with a lot of text generation both for a summarisation for machine translation
00:52:13
we really don't have the reader angle in there at all right so there's hardly anyone who cares
00:52:18
for the person who actually consume the stuff that you generate so we just say oh there's use keyword overlap
00:52:23
uh and the more about the better and the links should somewhat somewhat
00:52:27
work right so if you look at your blue scores and all these other
00:52:30
metrics for um generated text quality this there's never anything about the reader and
00:52:35
uh sometimes you have something like reading level but that's sort of it um
00:52:41
on the other hand you a few months to write texts and ideally if you
00:52:45
they do this well they have a very clear idea who will read this thing
00:52:48
what they already know and what they want to know from this text right so very very different picture not just
00:52:53
all we text generation engines that are pretty good hopefully but we also care about the readership so i think this
00:53:00
uh in general so for a bunch of these tasks like in our
00:53:04
new summarisation or your jaw summarisation idea here ah is going to be
00:53:09
very key in the next few years to look at user models or
00:53:12
at least audience models also groups of users the to get a half
00:53:15
similar uh um requirements and needs towards the text that you're generating there
00:53:20
okay i'm just too too close here so we look at three clinical problems
00:53:25
um i i hope with some properties that we see real career in many
00:53:29
many other a domain so that we're not just sort of uh solving certain
00:53:33
um technical tasks but really also i'll hopefully addressing
00:53:37
interesting an l. p. or i i problems um
00:53:41
and as the closing thought i i would think that there's so much enthusiasm about precision madison
00:53:46
but i'm actually physicians are really good at this already so we train them for precision and
00:53:50
they're they're they're really strong at this now recall the something that humans are not built for
00:53:55
right so considering seventy signals concurrently at second resolution nobody can do that
00:54:01
uh really quickly going and read a million medical articles and going
00:54:04
back to the patient and telling them what's going on people also
00:54:07
come to that so uh i would strongly advocate for something like
00:54:11
recall medicine right so we have the machinery how all precision schumann brain
00:54:16
uh on the recall part and say well look here's all the information that you want to
00:54:19
have now go into you shouldn't thing right so i think that's going to be really important um
00:54:25
so as an interesting fact here's also bunch of years ago off um people attribute to medical error on
00:54:32
soul decision making going wrong somewhere in the clinical pipeline right so leading to either the wrong diagnosis or treatment
00:54:37
um as the third leading cause of that right so that means you have cancer and you of congestive heart failure
00:54:43
uh and then before you have traffic before your gun violence before you have
00:54:47
all these other good things um you half megahertz right so if we can
00:54:51
use some of the uh the machinery that we all work on a to chip away on
00:54:56
this huge budget i think that may make much
00:54:59
bigger impact then um then that's holding individual diseases
Conference Program
Clinical Natural Language Processing
Carsten Eickhoff, Assistant professor of medical and computer science at Brown University
Aug. 20, 2019 · 11:05 a.m.
299 views
Clinical Natural Language Processing Q&A
Carsten Eickhoff, Assistant professor of medical and computer science at Brown University
Aug. 20, 2019 · noon
110 views
Recommended talks
AI and Radiology
Dr Amine Korchi
Feb. 18, 2019 · 11:05 a.m.
846 views
