Ranking annotators for crowdsourced labeling tasks
With the advent of crowdsourcing services it has become quite cheap and reasonably effective to get a dataset labeled by multiple annotators in a short amount of time. Various methods have been proposed to estimate the consensus labels by correcting for the bias of annotators with different kinds of expertise. Often we have low quality annotators or spammers--annotators who assign labels randomly (e.g., without actually looking at the instance). Spammers can make the cost of acquiring labels very expensive and can potentially degrade the quality of the consensus labels. In this paper we formalize the notion of a spammer and define a score which can be used to rank the annotators---with the spammers having a score close to zero and the good annotators having a high score close to one.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks
Supervised classification heavily depends on datasets annotated by humans. However, in subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. Annotations have common…
ClassificationFairnessParsimonious Mixed-Effects HodgeRank for Crowdsourced Preference Aggregation
In crowdsourced preference aggregation, it is often assumed that all the annotators are subject to a common preference or utility function which generates their comparison behaviors in experiments. However, in reality an…
Candidate Labeling for Crowd Learning
Crowdsourcing has become very popular among the machine learning community as a way to obtain labels that allow a ground truth to be estimated for a given dataset. In most of the approaches that use crowdsourced labels, …
Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification
Crowdsourcing systems have been used to accumulate massive amounts of labeled data for applications such as computer vision and natural language processing. However, because crowdsourced labeling is inherently dynamic an…
Computational EfficiencyEnsemble LearningBayesian Crowdsourcing with Constraints
Crowdsourcing has emerged as a powerful paradigm for efficiently labeling large datasets and performing various learning tasks, by leveraging crowds of human annotators. When additional information is available about the…
Variational Inference