Quality and Quantity of Machine Translation References for Automatic Metrics
Automatic machine translation metrics typically rely on human translations to determine the quality of system translations. Common wisdom in the field dictates that the human references should be of very high quality. However, there are no cost-benefit analyses that could be used to guide practitioners who plan to collect references for machine translation evaluation. We find that higher-quality references lead to better metric correlations with humans at the segment-level. Having up to 7 references per segment and taking their average (or maximum) helps all metrics. Interestingly, the references from vendors of different qualities can be mixed together and improve metric success. Higher quality references, however, cost more to create and we frame this as an optimization problem: given a specific budget, what references should be collected to maximize metric success. These findings can be used by evaluators of shared tasks when references need to be created under a certain budget.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Optimizing Statistical Machine Translation for Text Simplification
Most recent sentence simplification systems use basic machine translation models to learn lexical and syntactic paraphrases from a manually simplified parallel corpus. These methods are limited by the quality and quantit…
Machine TranslationSentenceText SimplificationTranslationModeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specifi…
Machine TranslationSentenceTranslationA Quality-based Active Sample Selection Strategy for Statistical Machine Translation
This paper presents a new active learning technique for machine translation based on quality estimation of automatically translated sentences. It uses an error-driven strategy, i.e., it assumes that the more errors an au…
Active LearningMachine TranslationSentenceSentiment Analysis+1A Post-Editing Dataset in the Legal Domain: Do we Underestimate Neural Machine Translation Quality?
We introduce a machine translation dataset for three pairs of languages in the legal domain with post-edited high-quality neural machine translation and independent human references. The data was collected as part of the…
Automatic Post-EditingMachine TranslationSentenceTranslationBLEU might be Guilty but References are not Innocent
The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems. This paper demonstrates that, while choice of metric is important, the nature of t…
DiversityMachine TranslationTranslation