DATScore: Evaluating Translation with Data Augmented Translations
The rapid development of large pretrained language models has revolutionized not only the field of Natural Language Generation (NLG) but also its evaluation. Inspired by the recent work of BARTScore: a metric leveraging the BART language model to evaluate the quality of generated text from various aspects, we introduce DATScore. DATScore uses data augmentation techniques to improve the evaluation of machine translation. Our main finding is that introducing data augmented translations of the source and reference texts is greatly helpful in evaluating the quality of the generated translation. We also propose two novel score averaging and term weighting strategies to improve the original score computing process of BARTScore. Experimental results on WMT show that DATScore correlates better with human meta-evaluations than the other recent state-of-the-art metrics, especially for low-resource languages. Ablation studies demonstrate the value added by our new scoring strategies. Moreover, we report in our extended experiments the performance of DATScore on 3 NLG tasks other than translation.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationLanguage ModelingLanguage ModellingMachine TranslationText GenerationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DiHuTra: a Parallel Corpus to Analyse Differences between Human Translations
This project aimed to design a corpus of parallel human translations (HTs) of the same source texts by professionals and students. The resulting corpus consists of English news and reviews source texts, their translation…
Machine TranslationTranslationPET: a Tool for Post-editing and Assessing Machine Translation
Given the significant improvements in Machine Translation (MT) quality and the increasing demand for translations, post-editing of automatic translations is becoming a popular practice in the translation industry. It has…
Machine TranslationSentenceTranslationEvaluating Optimal Reference Translations
The overall translation quality reached by current machine translation (MT) systems for high-resourced language pairs is remarkably good. Standard methods of evaluation are not suitable nor intended to uncover the many t…
Machine TranslationTranslationDo LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality…
Machine TranslationQuestion AnsweringReading ComprehensionSentence+1Removing Biases from Trainable MT Metrics by Using Self-Training
Most trainable machine translation (MT) metrics train their weights on human judgments of state-of-the-art MT systems outputs. This makes trainable metrics biases in many ways. One of them is preferring longer translatio…
Domain AdaptationMachine TranslationTranslation