Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax
This article presents the results we obtained on a complex annotation task (that of dependency syntax) using a specifically designed Game with a Purpose, ZombiLingo. We show that with suitable mechanisms (decomposition of the task, training of the players and regular control of the annotation quality during the game), it is possible to obtain annotations whose quality is significantly higher than that obtainable with a parser, provided that enough players participate. The source code of the game and the resulting annotated corpora (for French) are freely available.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningNatural Language InferenceSimilar Papers 제목 키워드 기반
A 3D Role-Playing Game for Abusive Language Annotation
Gamification has been applied to many linguistic annotation tasks, as an alternative to crowdsourcing platforms to collect annotated data in an inexpensive way. However, we think that still much has to be explored. Games…
Abusive LanguageIt's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation
Annotated data is prerequisite for many NLP applications. Acquiring large-scale annotated corpora is a major bottleneck, requiring significant time and resources. Recent work has proposed turning annotation into a game t…
AllWord Sense DisambiguationA Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels
Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We p…
Semantic Role LabelingCorpus Annotation through Crowdsourcing: Towards Best Practice Guidelines
Crowdsourcing is an emerging collaborative approach that can be used for the acquisition of annotated corpora and a wide range of other linguistic resources. Although the use of this approach is intensifying in all its k…
Domain AdaptationNatural Language InferenceOpinion MiningQuestion Answering+1Crowdsourcing as a preprocessing for complex semantic annotation tasks
This article outlines a methodology that uses crowdsourcing to reduce the workload of experts for complex semantic tasks. We split turker-annotated datasets into a high-agreement block, which is not modified, and a low-a…