Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data
Crowdsourcing offers a convenient means of obtaining labeled data quickly and inexpensively. However, crowdsourced labels are often noisier than expert-annotated data, making it difficult to aggregate them meaningfully. We present an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications. The predicted labels achieve a correlation of 0.594 with expert labels on our data, outperforming the best alternative aggregation method by 11.9{\%}. Our approach also outperforms the alternatives on third-party datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionSimilar Papers 제목 키워드 기반
Label Selection Approach to Learning from Crowds
Supervised learning, especially supervised deep learning, requires large amounts of labeled data. One approach to collect large amounts of labeled data is by using a crowdsourcing platform where numerous workers perform …
Bayesian Detector Combination for Object Detection with Crowdsourced Annotations
Acquiring fine-grained object detection annotations in unconstrained images is time-consuming, expensive, and prone to noise, especially in crowdsourcing scenarios. Most prior object detection methods assume accurate ann…
Objectobject-detectionObject DetectionIdentifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations
Recent works of opinion expression identification (OEI) rely heavily on the quality and scale of the manually-constructed training corpus, which could be extremely difficult to satisfy. Crowdsourcing is one practical sol…
Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images
Automated detection of electron dense deposits (EDD) in glomerular disease is hindered by the scarcity of high-quality labeled data. While crowdsourcing reduces annotation cost, it introduces label noise. We propose an a…
Active LearningLearning Ambiguity from Crowd Sequential Annotations
Most crowdsourcing learning methods treat disagreement between annotators as noisy labelings while inter-disagreement among experts is often a good indicator for the ambiguity and uncertainty that is inherent in natural …
NERPOSPOS Tagging