Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation
Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task. We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents. In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. Our findings emphasise the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs.
Code (1)
Tasks
Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation
In this paper, we reassess claims of human parity and super human performance in machine translation. Although these terms have already been discussed, as well as the evaluation protocols used to achieved these conclusio…
Machine TranslationTranslationAttaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation
We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three var…
Machine TranslationTranslationAchieving Human Parity on Automatic Chinese to English News Translation
Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question natu…
Machine TranslationTranslationReassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019
We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify…
Machine TranslationTranslationA Set of Recommendations for Assessing Human-Machine Parity in Language Translation
The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We rea…
Machine TranslationTranslation