DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit
Dialog system developers need high-quality data to train, fine-tune and assess their systems. They often use crowdsourcing for this since it provides large quantities of data from many workers. However, the data may not be of sufficiently good quality. This can be due to the way that the requester presents a task and how they interact with the workers. This paper introduces DialCrowd 2.0 to help requesters obtain higher quality data by, for example, presenting tasks more clearly and facilitating effective communication with workers. DialCrowd 2.0 guides developers in creating improved Human Intelligence Tasks (HITs) and is directly applicable to the workflows used currently by developers and researchers.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DialCrowd: A toolkit for easy dialog system assessment
When creating a dialog system, developers need to test each version to ensure that it is performing correctly. Recently the trend has been to test on large datasets or to ask many users to try out a system. Crowdsourcing…
ChatbotData Collection for Dialogue System: A Startup Perspective
Industrial dialogue systems such as Apple Siri and Google Now rely on large scale diverse and robust training data to enable their sophisticated conversation capability. Crowdsourcing provides a scalable and inexpensive …
General Classificationintent-classificationIntent ClassificationText ClassificationService registration chatbot: collecting and comparing dialogues from AMT workers and service’s users
Crowdsourcing is the go-to solution for data collection and annotation in the context of NLP tasks. Nevertheless, crowdsourced data is noisy by nature; the source is often unknown and additional validation work is perfor…
ChatbotText GenerationThe DialPort tools
The DialPort project http://dialport.org/, funded by the National Science Foundation (NSF), covers a group of tools and services that aim at fulfilling the needs of the dialog research community. Over the course of six y…
CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues
Large corpora of task-based and open-domain conversational dialogues are hugely valuable in the field of data-driven dialogue systems. Crowdsourcing platforms, such as Amazon Mechanical Turk, have been an effective metho…