Abstract
Disentangling conversations mixed together in
a single stream of messages is a difficult task,
made harder by the lack of large manually annotated datasets. We created a new dataset
of 77,563 messages manually annotated with
reply-structure graphs that both disentangle
conversations and define internal conversation
structure. Our dataset is 16 times larger than
all previously released datasets combined, the
first to include adjudication of annotation disagreements, and the first to include context.
We use our data to re-examine prior work, in
particular, finding that 80% of conversations in
a widely used dialogue corpus are either missing messages or contain extra messages. Our
manually-annotated data presents an opportunity to develop robust data-driven methods for
conversation disentanglement, which will help
advance dialogue research