Multi-Hypothesis Machine Translation Evaluation
Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. One of the main challenges is the fact that multiple outputs can be equally valid. Attempts to minimise this issue include metrics that relax the matching of MT output and reference strings, and the use of multiple references. The latter has been shown to significantly improve the performance of evaluation metrics. However, collecting multiple references is expensive and in practice a single reference is generally used. In this paper, we propose an alternative approach: instead of modelling linguistic variation in human reference we exploit the MT model uncertainty to generate multiple diverse translations and use these: (i) as surrogates to reference translations; (ii) to obtain a quantification of translation variability to either complement existing metric scores or (iii) replace references altogether. We show that for a number of popular evaluation metrics our variability estimates lead to substantial improvements in correlation with human judgements of quality by up 15{\%}.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationvalidSimilar Papers 제목 키워드 기반
Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model
We propose an automatic evaluation method of machine translation that uses source language sentences regarded as additional pseudo references. The proposed method evaluates a translation hypothesis in a regression model.…
Language ModelingLanguage ModellingMachine TranslationSentence+1Machine Translation Models are Zero-Shot Detectors of Translation Direction
Detecting the translation direction of parallel text has applications for machine translation training and evaluation, but also has forensic applications such as resolving plagiarism or forgery allegations. In this work,…
Machine TranslationNMTTranslationSampling and Filtering of Neural Machine Translation Distillation Data
In most of neural machine translation distillation or stealing scenarios, the highest-scoring hypothesis of the target model (teacher) is used to train a new model (student). If reference translations are also available,…
Machine TranslationTranslationRevisit Automatic Error Detection for Wrong and Missing Translation -- A Supervised Approach
While achieving great fluency, current machine translation (MT) techniques are bottle-necked by adequacy issues. To have a closer study of these issues and accelerate model development, we propose automatic detecting ade…
Machine TranslationTranslationSampling and Filtering of Neural Machine Translation Distillation Data
In most of neural machine translation distillation or stealing scenarios, the goal is to preserve the performance of the target model (teacher). The highest-scoring hypothesis of the teacher model is commonly used to tra…
Machine TranslationTranslation