A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation
The target outputs of many NLP tasks are word sequences. To collect the data for training and evaluating models, the crowd is a cheaper and easier to access than the oracle. To ensure the quality of the crowdsourced data, people can assign multiple workers to one question and then aggregate the multiple answers with diverse quality into a golden one. How to aggregate multiple crowdsourced word sequences with diverse quality is a curious and challenging problem. People need a dataset for addressing this problem. We thus create a dataset (CrowdWSA2019) which contains the translated sentences generated from multiple workers. We provide three approaches as the baselines on the task of extractive word sequence aggregation. Specially, one of them is an original one we propose which models the reliability of workers. We also discuss some issues on ground truth creation of word sequences which can be addressed based on this dataset.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
Identifying and assigning keywords at scale is a technical, practical, and ethical challenge for crowdsourced collections. This article reports the findings of the "Extracting Keywords from Crowdsourced Collections" proj…
Keyword ExtractionAsking questions on handwritten document collections
This work addresses the problem of Question Answering (QA) on handwritten document collections. Unlike typical QA and Visual Question Answering (VQA) formulations where the answer is a short text, we aim to locate a docu…
Optical Character Recognition (OCR)Question AnsweringvalidVisual Question Answering+1Language Models as Context-sensitive Word Search Engines
Context-sensitive word search engines are writing assistants that support word choice, phrasing, and idiomatic language use by indexing large-scale n-gram collections and implementing a wildcard search. However, search r…
Language ModelingLanguage ModellingShallow pooling for sparse labels
Recent years have seen enormous gains in core IR tasks, including document and passage ranking. Datasets and leaderboards, and in particular the MS MARCO datasets, illustrate the dramatic improvements achieved by modern …
Passage RankingFocal Visual-Text Attention for Visual Question Answering
Recent insights on language and vision with neural networks have been successfully applied to simple single-image visual question answering. However, to tackle real-life question answering problems on multimedia collecti…
Memex Question AnsweringQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)