Crowdsourcing for Evaluating Machine Translation Quality
The recent popularity of machine translation has increased the demand for the evaluation of translations. However, the traditional evaluation approach, manual checking by a bilingual professional, is too expensive and too slow. In this study, we confirm the feasibility of crowdsourcing by analyzing the accuracy of crowdsourcing translation evaluations. We compare crowdsourcing scores to professional scores with regard to three metrics: translation-score, sentence-score, and system-score. A Chinese to English translation evaluation task was designed using around the NTCIR-9 PATENT parallel corpus with the goal being 5-range evaluations of adequacy and fluency. The experiment shows that the average score of crowdsource workers well matches professional evaluation results. The system-score comparison strongly indicates that crowdsourcing can be used to find the best translation system given the input of 10 source sentence.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
APE-QUEST: an MT Quality Gate
The APE-QUEST project (2018--2020) sets up a quality gate and crowdsourcing workflow for the eTranslation system of EC’s Connecting Europe Facility to improve translation quality in specific domains. It packages these se…
TranslationExperiences in Resource Generation for Machine Translation through Crowdsourcing
The logistics of collecting resources for Machine Translation (MT) has always been a cause of concern for some of the resource deprived languages of the world. The recent advent of crowdsourcing platforms provides an opp…
Machine TranslationTranslationUsing crowdsourcing system for creating site-specific statistical machine translation engine
A crowdsourcing translation approach is an effective tool for globalization of site content, but it is also an important source of parallel linguistic data. For the given site, processed with a crowdsourcing system, a se…
Machine TranslationSentenceTranslationData Quality in Crowdsourcing and Spamming Behavior Detection
As crowdsourcing emerges as an efficient and cost-effective method for obtaining labels for machine learning datasets, it is important to assess the quality of crowd-provided data, so as to improve analysis performance a…
Face VerificationPerceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation
The quality of machine-generated text is a complex construct consisting of various aspects and dimensions. We present a study that aims to uncover relevant perceptual quality dimensions for one type of machine-generated …
AttributeMachine TranslationTranslation