A Neural Model for Aggregating Coreference Annotation in Crowdsourcing
Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a high-performance coreference resolution system is time-consuming and expensive. Crowdsourcing makes it possible to obtain the required amounts of data rapidly and cost-effectively. However, crowd-sourced labels can be noisy. To ensure high-quality data, it is crucial to infer the correct labels by aggregating the noisy labels. In this paper, we split the aggregation into two subtasks, i.e, mention classification and coreference chain inference. Firstly, we predict the general class of each mention using an autoencoder, which incorporates contextual information about each mention, while at the same time taking into account the mention{'}s annotation complexity and annotators{'} reliability at different levels. Secondly, to determine the coreference chain of each mention, we use weighted voting which takes into account the learned reliability in the first subtask. Experimental results demonstrate the effectiveness of our method in predicting the correct labels. We also illustrate our model{'}s interpretability through a comprehensive analysis of experimental results.
Code (0)
등록된 구현이 없습니다.
Tasks
coreference-resolutionCoreference ResolutionSimilar Papers 제목 키워드 기반
A Probabilistic Annotation Model for Crowdsourcing Coreference
The availability of large scale annotated corpora for coreference is essential to the development of the field. However, creating resources at the required scale via expert annotation would be too expensive. Crowdsourcin…
Coreference ResolutionmodelQuestion AnsweringCrowdsourcing and Aggregating Nested Markable Annotations
One of the key steps in language resource creation is the identification of the text segments to be annotated, or markables, which depending on the task may vary from nominal chunks for named entity resolution to (potent…
coreference-resolutionCoreference ResolutionEntity ResolutionText SegmentationCoRefi: A Crowd Sourcing Suite for Coreference Annotation
Coreference annotation is an important, yet expensive and time consuming, task, which often involved expert annotators trained on complex decision guidelines. To enable cheaper and more efficient annotation, we present C…
CoRefi: A Crowd Sourcing Suite for Coreference Annotation
Coreference annotation is an important, yet expensive and time consuming, task, which often involved expert annotators trained on complex decision guidelines. To enable cheaper and more efficient annotation, we present C…
Coreference ResolutionCross Document Coreference ResolutionA Mention-Pair Model of Annotation with Nonparametric User Communities
The availability of large datasets is essential for progress in coreference and other areas of NLP. Crowdsourcing has proven a viable alternative to expert annotation, offering similar quality for better scalability. How…