Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric--conditioning on the source instead of the reference--and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.
Code (1)
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource lan…
Machine TranslationTranslationInvestigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study
Machine Translation has made impressive progress in recent years offering close to human-level performance on many languages, but studies have primarily focused on high-resource languages with broad online presence and r…
Machine TranslationTransfer LearningTranslationAn Automatic Evaluation of the WMT22 General Machine Translation Task
This report presents an automatic evaluation of the general machine translation task of the Seventh Conference on Machine Translation (WMT22). It evaluates a total of 185 systems for 21 translation directions including h…
Machine TranslationTranslationAssessing Evaluation Metrics for Speech-to-Speech Translation
Speech-to-speech translation combines machine translation with speech synthesis, introducing evaluation challenges not present in either task alone. How to automatically evaluate speech-to-speech translation is an open q…
Machine TranslationOpen-Ended Question AnsweringSpeech SynthesisSpeech-to-Speech Translation+1Aligning Word Vectors on Low-Resource Languages with Wiktionary
Aligned word embeddings have become a popular technique for low-resource natural language processing. Most existing evaluation datasets are generated automatically from machine translations systems, so they have many err…
Word Embeddings