Aggregating and Predicting Sequence Labels from Crowd Annotations
Despite sequences being core to NLP, scant work has considered how to handle noisy sequence labels from multiple annotators for the same text. Given such annotations, we consider two complementary tasks: (1) aggregating sequential crowd labels to infer a best single set of consensus annotations; and (2) using crowd annotations as training data for a model that can predict sequences in unannotated text. For aggregation, we propose a novel Hidden Markov Model variant. To predict sequences in unannotated text, we propose a neural approach using Long Short Term Memory. We evaluate a suite of methods across two different applications and text genres: Named-Entity Recognition in news articles and Information Extraction from biomedical abstracts. Results show improvement over strong baselines. Our source code and data are available online.
Code (1)
Tasks
Articlesnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Part-Of-Speech TaggingSimilar Papers 제목 키워드 기반
A Bayesian Approach for Sequence Tagging with Crowds
Current methods for sequence tagging, a core task in NLP, are data hungry, which motivates the use of crowdsourcing as a cheap way to obtain labelled data. However, annotators are often unreliable and current aggregation…
Active LearningArgument Miningnamed-entity-recognitionNamed Entity Recognition+1Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization
Selecting an effective training signal for tasks in natural language processing is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. At the same time, recent work in NLP has …
Domain GeneralizationKnowledge DistillationA Neural Model for Aggregating Coreference Annotation in Crowdsourcing
Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a high-performance coreference resolu…
coreference-resolutionCoreference ResolutionAggregating and Learning from Multiple Annotators
The success of NLP research is founded on high-quality annotated datasets, which are usually obtained from multiple expert annotators or crowd workers. The standard practice to training machine learning models is to firs…
Modeling sequential annotations for sequence labeling with crowds
Crowd sequential annotations can be an efficient and cost-effective way to build large datasets for sequence labeling. Different from tagging independent instances, for crowd sequential annotations the quality of label s…
valid