Analysis of Automatic Annotation Suggestionsfor Hard Discourse-Level Tasks in Expert Domains
Abstract
Many complex discourse-level tasks can aid
domain experts in their work but require costly
expert annotations for data creation. To speed
up and ease annotations, we investigate the viability of automatically generated annotation
suggestions for such tasks. As an example,
we choose a task that is particularly hard for
both humans and machines: the segmentation and classification of epistemic activities
in diagnostic reasoning texts. We create and
publish a new dataset covering two domains
and carefully analyse the suggested annotations. We find that suggestions have positive
effects on annotation speed and performance,
while not introducing noteworthy biases. Envisioning suggestion models that improve with
newly annotated texts, we contrast methods for
continuous model adjustment and suggest the
most effective setup for suggestions in future
expert tasks