Document-Level Machine Translation Evaluation Project: Methodology, Effort and Inter-Annotator Agreement
Document-level (doc-level) human eval-uation of machine translation (MT) has raised interest in the community after a fewattempts have disproved claims of “human parity” (Toral et al., 2018; Laubli et al.,2018). However, little is known about bestpractices regarding doc-level human evalu-ation. The goal of this project is to identifywhich methodologies better cope with i)the current state-of-the-art (SOTA) humanmetrics, ii) a possible complexity when as-signing a single score to a text consisted of‘good’ and ‘bad’ sentences, iii) a possibletiredness bias in doc-level set-ups, and iv)the difference in inter-annotator agreement(IAA) between sentence and doc-level set-ups.
Code (0)
등록된 구현이 없습니다.
Tasks
Document Level Machine TranslationMachine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
DELA Project: Document-level Machine Translation Evaluation
This paper presents the results of the DELA Project. We describe the testing of context span for document-level evaluation, construction of a document-level corpus, and context position, as well as the latest development…
Document Level Machine TranslationMachine TranslationPositionTranslationHuman and Automatic Evaluation of English-Hindi Machine Translation
For the past 60 years, Research in machine translation is going on. For the development in this field, a lot of new techniques are being developed each day. As a result, we have witnessed development of many automatic ma…
Machine TranslationSentenceTranslationDiscourse Cohesion Evaluation for Document-Level Neural Machine Translation
It is well known that translations generated by an excellent document-level neural machine translation (NMT) model are consistent and coherent. However, existing sentence-level evaluation metrics like BLEU can hardly ref…
Machine TranslationNMTSentenceTranslationHas Machine Translation Achieved Human Parity? A Case for Document-level Evaluation
Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task. We empirically test this claim with alternative evaluation p…
Machine TranslationSentenceTranslationOn the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation
Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of “human parity” (Toral et al., 2018; Läubli et al., 2018) with document-level human evaluat…
Machine TranslationSentenceTranslation