Player is loading...
Automated Fact-checking: an NLP perspective
Prof. Andreas Vlachos, University Cambridge
Wednesday, Feb. 21, 2024 · 9:45 a.m. · 21m 10s
Misinformation is considered one of the major challenges of our times resulting in numerous efforts against it. Fact-checking, the task of assessing whether a claim is true or false, is considered a key weapon in reducing its impact. In this talk I will present our recent and ongoing work on automating this task using natural language processing focusing on the importance of accompanying verdicts with evidence and justifications.
Embed
Transcriptions
Note: this content has been automatically generated.
00:00:00
uh hello they remotes with limitations which are coming here they ah yes so
00:00:06
fact checking i think many of your we're given that you can be spent
00:00:10
uh it's a a it it's a city away that would
00:00:14
take the part is also media have couple of information and misinformation
00:00:18
spread more up with them before however what has not saying is
00:00:22
is that the mind of fox can take hours or even days
00:00:26
and uh she is a marvel talking when the seasons around the world actually there's a lot of people working on it
00:00:32
but uh i think we would all agree there's not a not a and
00:00:36
um so just make things were great what do they do um something like this
00:00:40
for example what because they claim about the the number of redundancy in either the
00:00:44
the net kingdom and then those organisations box you that would actually take this pain
00:00:50
compared to the data that time the claim was made so that's a while ago and in this case you conclude that yeah this was reporting
00:00:56
and the given explanation yeah there's no way you can find ten times a it
00:01:01
either this number of interest in the u. k. and the automated fuck techniques to be
00:01:05
as desperately it that if you like the aspiration here is that we can have
00:01:09
a sort of conformity feedback that would actually be able to do this work for us
00:01:13
and that let the packages with with this in like right top people that's or
00:01:18
and then now uh all of this was actually sort
00:01:22
of in inspired by politicians saying things to the people
00:01:26
and maybe pundits and so on but that that they actually knew whither items are the very i might have but
00:01:32
since it became a one to check out a sensibility and the in that person's well you know what if you actually
00:01:39
it's a bit inside of me that form other orange that's right component and that was a bit
00:01:44
of that later um so what i said we're gonna zack that was about a decade ago um
00:01:50
well at the show was actually computer vision of actually getting that much better you cover at the
00:01:56
time how onyx net and you know exactly actually have a exactly sprinkled yep these that the cat
00:02:02
and that however there was criticism that that you know that's
00:02:06
secure program actually c. d. has a very nice uh strip where
00:02:09
says well you know yeah so uh is your much learning system yeah you put data in this big pile of linear algebra
00:02:15
yeah they like that's all the other side the what if it's wrong
00:02:18
while it just started it and really get the right is and like uh
00:02:22
swordfish explain integrating centrifuges you know about it and then
00:02:26
yeah that works but couldn't businesses factor right so so
00:02:31
that's why one of the things that we didn't think it was the one who was from limits for signal
00:02:35
first thing i would like to say is everything it's right if someone tells you that this is false or true like demurely any of it
00:02:42
as well i think we shouldn't that take it no mother put that
00:02:45
that thing is it could be machine could be schumann could be newspapers on
00:02:49
and that i would also the labels along the look and you should too far
00:02:53
base this tried source as they you know even if say that happen to be
00:02:56
true but if so people start grandpa charting that's falls that's true doesn't matter right
00:03:01
the the point that you would have downloaded society that's the foundational aspect of the morris
00:03:07
and then i know yeah say from the plane to people like myself but
00:03:10
the systems is that that if i just sacked the credit of the fox
00:03:14
it's much easier for me to actually the somewhat would probably my system if i
00:03:18
get to the end of this infusion of to the something which a portion to
00:03:22
uh and i will watch to be able to learn without the maple beta uh this say one of
00:03:27
the successes in uh uh of notions processing has been all the markings interface wants now is quite usable
00:03:33
uh however i was a little bit of very large corporate like the part of the proceedings in mainland was of the european parliament
00:03:39
now as i said the one the five second take up our sort of they start to produce so wanted that scale
00:03:46
and then enter the although i like to say uh uh in
00:03:50
this this that that we should be thinking about its intended use
00:03:54
and that also beware or somebody that's known as the white hot
00:03:58
bias as any okay ah above the classifier i mean the depression above
00:04:02
your class part for breakage fantastic you know how would that be
00:04:06
doing something bob wasn't shaggy right above that that's part that fake news
00:04:10
however if i think about actually thing about well okay yeah but
00:04:14
the class for give me label but is actually shopping in one
00:04:17
uh when i use it was that that's going to be fox it was going to want the best for the pictures that
00:04:22
they want to read the part sixty but just give neighbours exactly be is a good which output is gonna be use that
00:04:28
to assess the uh put the professorship i'd say yeah this you're gonna say it it would be the the functions of media problem
00:04:35
so it's more the something about these things and uh we've got the discussion of these issues
00:04:40
one the first paper intent courting as well as a more recent paper and pen to three
00:04:45
um so when it comes to making it um that pipeline
00:04:49
for parting can go like this house is the came detection
00:04:53
that's an important part uh because uh one of the keys passing the part second
00:04:58
question was deciding what the fox it because then the spend a day doing it
00:05:02
so it's important to dwell in the first instance and then every this with three well
00:05:06
uh that's what sounds in because we need to search the web and a
00:05:10
part or other sources of called up experts all sort of things that you do
00:05:14
and then then you have this is a very big production deciding that would some is supported
00:05:19
by evidence event justification productions wednesday was just the
00:05:23
labels but also so why that labels are given
00:05:26
and we've covered it in the cup of surveys on the actual only five of
00:05:29
them it was taking a more simple on them with a model parts taking um
00:05:34
now one thing that became apparent to me and i think a lot of people would agree is that that you know
00:05:40
to how a breakthrough in the artificial intelligence uh you need to have the i guess it's not just how good some models
00:05:46
and that there's a lot to put quite nicely by someone that all the stuff also look at the block
00:05:52
was working at the stable welcome might disagree or was
00:05:56
a breakthrough in a i what would some shame about promotional
00:05:59
but the messengers here that you know in order to actually have say some probably some substantial problems in a i
00:06:06
you'd have a data to for it not just the outward and um for this
00:06:10
purpose we present the the the facts identification that that's at the where we actually had
00:06:15
the uh other this provision claims this one about the rodney king riots and then we
00:06:20
had a whole day those other vehicles would technically and go back to the big yeah
00:06:25
find that with this and then decide whether the evidence was supporting or including
00:06:28
the evidence or but just couldn't put enough information for this purpose and the
00:06:34
and that was a little had them we were able to construct the burden that isn't that what the five thousand claims
00:06:39
and that in the evaluation what we had was that it was not enough to
00:06:44
give us the right label info you have to give us the right evidence to
00:06:48
correct label not quite evidence woodrow no points only
00:06:52
further identify this as a getting some pointing and actors
00:06:56
and uh okay so i was nice right do we have some evidence and
00:07:00
that made if there's more popular as a uh does it about them but that
00:07:05
that is i don't think there's just the baseline right you know okay so here's
00:07:09
the it was possible but with michael's want to know how the this using recent verdict
00:07:15
or in the what was what were the assumptions or the commonsense
00:07:18
shoes and then in more broadly what was the reasoning person the model
00:07:22
and the current approaches mostly for some i there's a it's a using highlights
00:07:26
is the um there's notation megan's more than the the neural networks that are
00:07:30
applied to nachos processing uh sources you kind of say as the model to
00:07:34
give you okay tell me which was tokens which words you poke some more
00:07:39
however when people investigate that uh they realised that that i can choose not
00:07:44
to as a um a is no that's certainly been explanation and that that
00:07:48
there's nothing because maybe then called the pensions not an explanation in the patient
00:07:52
being not not an explanation and so you can see that it's it's another debate
00:07:57
another aspect was that then uh i don't think that's indicate how about if i
00:08:00
give you a very long list of which by this okay then no summarised for you
00:08:04
and that and then we'll covers is the reasoning however then it's not clear
00:08:09
whether this summer to get correspond to the vision in process of the model
00:08:13
what's causing both cases in fact is something that that sex is the for the space you want the justification to be
00:08:20
faithful to that models working so and that's a problem here like neither of these options actually
00:08:25
all products that because it's not clear they could say is this justification we get the different right
00:08:31
for this but there is a we proposed a different approach
00:08:33
when you get a proof work and then his systematically presentation
00:08:39
so probably playing uh such as this one about that because it would be useful sorry but serves the cans the have
00:08:44
the evidence and the no what we do is we actually
00:08:48
allowing parts of the claim would part of the evidence and we
00:08:53
put an operator needs alignments will for the beginning with with
00:08:57
with you say it's equivalent to the to do that's fun
00:09:00
the evidence with the short story we align it too is a noble and now in this case we say that these are
00:09:06
two different things visible alternately alternatives that are not compatible and
00:09:11
that's on the results of the normal beagle and there's also
00:09:14
is short and then the last part is uh when i'm charles dickens will styles they seek is now isn't it more challenging
00:09:21
because you have got to figure that okay there's a difference in the cow you you wrote the name however modern an l.
00:09:27
p. actually is good enough for this purpose so we can get the alignment correctly in say that yeah these the same person
00:09:33
and then now this is pushed to a some cold anatomical as i say it's
00:09:38
it doesn't get the good a set of rules in digital form either with these
00:09:41
where essentially um we start at the supporting state and that's a green state and then if we people
00:09:48
who stay there are however we'll have in all the good thing uh you got to be shooting the art
00:09:53
and this is this means now that you are the the evidence is if you think that the
00:09:59
and now when you're ugly with and then well basically the but you're the should do so you cock stay there
00:10:04
and from that will come up on us as they infer that this was actually a claim that we should by the evidence
00:10:11
and this is actually a face full thing because every time something changes in the operators
00:10:15
you have a you have to go through the common goal and then you get a different that the a different the verdict possibly
00:10:22
and then the six operators i actually made the previous work which has a long tradition in the
00:10:27
in it because of the line was and logic them on this not
00:10:31
for logic and they include tournaments negation like we've been so that nation
00:10:35
um and now this but because of that but actually what what happened exactly that that there was a
00:10:40
way of not turning this process into question answering so
00:10:43
no sort of actually having some that assigns operators directly
00:10:47
what we actually do with the with the claims as this one but on rice being more new jersey and then we ask questions about
00:10:53
on rice is other partners of rising get yeah so i guess they
00:10:56
can assign the operator could but and then rest my rise more will
00:11:01
arise when you just say that the paraphrase and so on and
00:11:04
they have different that questions that core different operator so for example you
00:11:08
look for the nation you might have here doesn't adjusting to new orleans
00:11:13
the answer is no so then you can actually get the different uh
00:11:16
uh so you're gonna see that this is not the word to be supporting of the cup
00:11:21
it's the new jersey is not new orleans so it's two separate things we'll get to that should be
00:11:25
but that was interesting here is that now thanks to the problems in language model being
00:11:30
any particular how there appears for question answering were able to turn this pursuit of having people
00:11:35
and actually coming up with ways of generating data sets with the operators will but i described is
00:11:40
that we can actually use a uh that that was most of the question answering second do reasonably well
00:11:45
you get these operators meatballs still through this will come up on that is the same upon what was before they could get a bird
00:11:52
and the what's even more exciting to me without actually knowing how was questioning with a language
00:11:57
was we can give the question in interviews and the premise that the good that thing with the
00:12:02
everything danish a we can get the answer i thought that's what would be the would ponder
00:12:05
this work better than actually just their devastating the data between the language and randy we reason system
00:12:11
and then that was some solos problem but i think there's also my point that this this it says there's room for improvement
00:12:17
on this is just the beginning my mind of having is is the most rational way to do uh the fact signal dramatically
00:12:24
and that can give us a as well handle all having but the situations that's
00:12:28
more interpret double and see where the models were all all of this what i
00:12:34
described essentially based on data that with the the the the we could be one
00:12:37
people watch the usual don't claim someone's also women could but what about the real world
00:12:42
these don't look like the kind of things that are the sparks so and then
00:12:47
why why the buildings because uh um this may the course actually because it's easier
00:12:53
um but uh we could uh what's was how cool successes we keeping the up and that the
00:12:58
problem with the real well they this is about been around for a while because exist people can
00:13:02
be they can say oh couple that they completely fucked because you're from what source or snow it's
00:13:06
and i make it a sort of that that's fine you can do that and people have done that
00:13:10
however the problems that when they look for evidence they okay well i'll put it this they give it to people at the top ten
00:13:16
sheets that's the evidence about how how do we know that the top
00:13:22
ten sheets of who are indeed the aim is used for fox taking
00:13:26
in fact the notes because that's not the case but open the and this is not in the top ten digits
00:13:31
um and the other part that uh when they did that they've been actually had the if there would be sure that little using the
00:13:37
fox taking pace for the play as part of the evidence but
00:13:40
the point what it have already begun the partaking article for the play
00:13:45
why then either the movies boxing system i'd of done the work right
00:13:48
um so for this reason we actually presenting in the circle to protect
00:13:53
a ensures what would be good to good thing no these are claims that the work prospect by fox checkers
00:13:59
and basically review it be i provide valuable that actually gives us a a nice way of collecting them because the party submit them there
00:14:05
and we have done this for the dates and uh we actually admitted they that's as references uh
00:14:12
speakers or swim uh if someone says oh you know how hard it
00:14:16
has been better last year which was the country being talked about that song
00:14:20
and then we hadn't it goes right questions and answers based on the partaking article so
00:14:26
the point was that well enough with the composed the farting portions the kind of question answering
00:14:32
and they had to not just give us the answers so they were inspire the five by the farting
00:14:37
article but to give us the answers they can't find the answer as we're not on the parting article
00:14:43
so we give them access to which is a modified way p. i. where they were only about parsers
00:14:48
for us as a not because it appeared before the claims made sure they couldn't get things that happened afterwards
00:14:54
right and the also these we are archiving on our credible
00:14:58
just basically because by um what basis say insane disappears will show
00:15:03
actually to the store and record about so big and one of them for later proposal when they have some people just be the
00:15:10
and then we have the forward classification scheme where says involve supported that if you did
00:15:14
not enough evidence and but we had bought the uh fourth option which is like
00:15:19
they can thinking yeah but this or is being so when i'm just with the p. d. was very nice well we did
00:15:23
we did not bear for their contributions are that was quite rare we made the claims of ourselves all that little fine right
00:15:30
but now when you look for a as a for a part isn't real well when you find
00:15:34
that the s. and then you have you have a clean and you have evidence that points to
00:15:38
both directions and our take with that we don't want to actually decided for also for the partake
00:15:44
is what we should be saying we want to do is not to say look there's complete new evidence
00:15:49
then it's up to the journalist or the user more broad of the
00:15:51
system to decide how to communicate that to the people but we think
00:15:56
that what they're partaking model should do that to present both sides of
00:16:00
the evidence and that that you will decide the user in this case
00:16:04
and um was provides justifications how did this come buys you the label
00:16:08
and then another process the we started with that loser say p. i. but
00:16:14
what we do is actually we don't just the diva the clay in
00:16:18
the lives of peace in what we do is actually compare larson was model
00:16:22
on whom i guess you're not aware of it isn't actually in it's an open part sticking more those
00:16:27
constructed back which was to move european countries and that when all the data that was used to train it
00:16:33
um i mean how it's uh well it's pretty accessible and
00:16:37
that it's equivalent to typically three i would say there's performance
00:16:41
and we use that to generate questions for the clean
00:16:45
and then getting those will get the evidence and now for a special band this we get
00:16:50
uh out of the top thirty pages we we you break it up in paragraphs and then we generate questions for its
00:16:56
part of and that's essentially the point of that that often
00:17:00
the way something's phrase in the evidence the weights reason to claim
00:17:04
they're not really good company that might use a party different words of some words might be implied
00:17:09
so by asking the questions over the power off of the evidence we kind of
00:17:12
say if you had that part of what questions would you be able to answer
00:17:17
and that this is hips breeds the gap between the two and um and finally i the rasta
00:17:23
production um we actually combine these in or the with it now with the questions and the answers
00:17:29
uh from anywhere with some of the claim and then we have uh as
00:17:32
as the uh classification scheme where if it's mixed get complete instead of picking
00:17:36
it's one supported really um which also what if we did otherwise it's not nothing it's and then i want to
00:17:44
uh take much more your time with results i would actually that the end is a three person more sizing part
00:17:49
and talk to people tried it it's not very good at giving the getting it tried to
00:17:54
give us the answers but this without asking questions i'm not much better than the the open model
00:18:00
uh and then bury my that we always consider the grass this isn't
00:18:04
the correctness of the label that that but the only given sufficient evidence
00:18:08
i believe in evaluating whether the evidence is correct uh is by the people because
00:18:14
the same it is come upon the main different web pages and main different praising so
00:18:17
that's working problems and the getting to that um you know one thing we haven't worked
00:18:23
on but i think it's very borders and distrust working has to be taken into account
00:18:27
a claim detection prayer this isn't hotter some work on that now that's a hopefully would be probably some um
00:18:34
we want to work with human fox checkers we actually one of the things we want to do is
00:18:37
to see so five hard trying to porsche whole could you write the partaking process versions immunity to like
00:18:43
actually we can actually make this part of the work so that that many have system adviser work a bit more
00:18:49
and then step just developed a sensible that's all hoping that probably win win situation
00:18:54
and we're going to have a fox second shirt that's for the next possession verification workshop
00:18:59
based on the separate big bits of real world fox ex uh if you want no more keeping up a good
00:19:04
question she would be at the minute b. which would be in november the it would would announce it next month
00:19:09
and there was want to expand to other inputs so i put on the about text but there's images and video want
00:19:15
to have to start the fast uh uh that's about as was uh uh so that was beginning this include my all
00:19:22
and uh i know i'm almost out of time so i want to tell
00:19:26
you uh i would be very quick on that and as i say that
00:19:29
it was mentioned there that the misinformation social event and there may if
00:19:34
there's anything that both increased polarisation but also deposit see also the bubble
00:19:38
so we'll have a project only as a finding that developing a sense that would help us uh
00:19:42
enhance deliberation begins humans so avoid this it gets
00:19:46
more like the dark wizard in division democratic reason
00:19:50
and then we have a data so that would release the word based on
00:19:53
it as a task that was the by by say the best bicycle this
00:19:57
and they're watching this thing we collect data and the some of the most insane is that when people started
00:20:04
they get it right eleven percent of the time after talking to each other they get rapid three percent of the time and what's
00:20:11
interesting is that important to personal the groups of people that have
00:20:15
the correct solution in the end not what is behind it recognition it
00:20:18
sure to list them other people complain since other i'm right in your own therefore there's two nice it
00:20:23
actually people shape of the southern get it right yes and um and that it's time to get it right so
00:20:30
if you want no more at or the so we get conversation
00:20:34
just getting them to speak longer that's not good enough it correlates
00:20:38
was the but very weakly there there's one usually solutions actually it's helpful even if they
00:20:42
don't have the right if they have the planning stage is more important ends actually but chapel
00:20:48
and the how do i think help them improve the burst of exclusions discussed actually is a flexible that
00:20:54
why they say something it expand to different solutions and i guess although this was not going to be possible
Conference Program
Opening and introduction
Prof. Lonneke van der Plas, Group Leader at Idiap, Computation, Cognition & Language
Feb. 21, 2024 · 9 a.m.
102 views
