Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and its relation with Direct Assessment
In this paper we present an analysis of the two most prominent methodologies used for the human evaluation of MT quality, namely evaluation based on Post-Editing (PE) and evaluation based on Direct Assessment (DA). To this purpose, we exploit a publicly available large dataset containing both types of evaluations. We first focus on PE and investigate how sensitive TER-based evaluation is to the type and number of references used. Then, we carry out a comparative analysis of PE and DA to investigate the extent to which the evaluation results obtained by methodologies addressing different human perspectives are similar. This comparison sheds light not only on PE but also on the so-called reference bias related to monolingual DA. Also, we analyze if and how the two methodologies can complement each other’s weaknesses.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSimilar Papers 제목 키워드 기반
A Set of Recommendations for Assessing Human-Machine Parity in Language Translation
The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We rea…
Machine TranslationTranslationFurther Investigation into Reference Bias in Monolingual Evaluation of Machine Translation
Monolingual evaluation of Machine Translation (MT) aims to simplify human assessment by requiring assessors to compare the meaning of the MT output with a reference translation, opening up the task to a much larger pool …
Machine TranslationTranslationStatistical Power and Translationese in Machine Translation Evaluation
The term translationese has been used to describe features of translated text, and in this paper, we provide detailed analysis of potential adverse effects of translationese on machine translation evaluation. Our analysi…
Machine TranslationTranslationEvaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation
Human evaluation is a critical component in machine translation system development and has received much attention in text translation research. However, little prior work exists on the topic of human evaluation for spee…
Machine TranslationTranslationBeyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive inves…
Machine TranslationTranslation