Do you know that Florence is packed with visitors?
Evaluating state-of-the-art models of speaker commitment
Abstract
When a speaker, Mary, asks Do you know that
Florence is packed with visitors?, we take her
to believe that Florence is packed with visitors,
but not if she asks Do you think that Florence
is packed with visitors? Inferring speaker
commitment (aka event factuality) is crucial
for information extraction and question answering. Here we explore the hypothesis that
linguistic deficits drive the error patterns of
speaker commitment models by analyzing the
linguistic correlates of model errors on a challenging naturalistic dataset. We evaluate two
state-of-the-art speaker commitment models
on the CommitmentBank, an English dataset
of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement (Florence is packed with visitors in our
example) of clause-embedding verbs (know,
think) under four entailment-canceling environments. We found that a linguisticallyinformed model outperforms a LSTM-based
one, suggesting that linguistic knowledge is
needed to capture such challenging naturalistic data. A breakdown of items by linguistic
features reveals asymmetrical error patterns:
while the models achieve good performance
on some classes (e.g., negation), they fail to
generalize to the diverse linguistic constructions (e.g., conditionals) in natural language,
highlighting directions for improvement