Crowdsourcing as a preprocessing for complex semantic annotation tasks
This article outlines a methodology that uses crowdsourcing to reduce the workload of experts for complex semantic tasks. We split turker-annotated datasets into a high-agreement block, which is not modified, and a low-agreement block, which is re-annotated by experts. The resulting annotations have higher observed agreement. We identify different biases in the annotation for both turkers and experts.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation
This paper describes a crowdsourcing experiment on the annotation of plot-like structures in English news articles. CrowdThruth methodology and metrics have been applied to select valid annotations from the crowd. We fur…
ArticlesRelationvalidA Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels
Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We p…
Semantic Role LabelingCrowdsourcing of Real-world Image Annotation via Visual Properties
Recent advances in data-centric artificial intelligence highlight inherent limitations in object recognition datasets. One of the primary issues stems from the semantic gap problem, which results in complex many-to-many …
Object RecognitionControlled Crowdsourcing for High-Quality QA-SRL Annotation
Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained p…
Semantic Role LabelingVocal Bursts Intensity PredictionsloWCrowd: A crowdsourcing tool for lexicographic tasks
The paper presents sloWCrowd, a simple tool developed to facilitate crowdsourcing lexicographic tasks, such as error correction in automatically generated wordnets and semantic annotation of corpora. The tool is open-sou…