The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation
Supervised Neural Machine Translation (NMT) systems currently achieve impressive translation quality for many language pairs. One of the key features of a correct translation is the ability to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. Existing evaluation benchmarks on WSD capabilities of translation systems rely heavily on manual work and cover only few language pairs and few word types. We present MuCoW, a multilingual contrastive test suite that covers 16 language pairs with more than 200 thousand contrastive sentence pairs, automatically built from word-aligned parallel corpora and the wide-coverage multilingual sense inventory of BabelNet. We evaluate the quality of the ambiguity lexicons and of the resulting test suite on all submissions from 9 language pairs presented in the WMT19 news shared translation task, plus on other 5 language pairs using NMT pretrained models. The MuCoW test suite is available at http://github.com/Helsinki-NLP/MuCoW.
Code (1)
Tasks
Machine TranslationNMTSentenceTranslationWord Sense DisambiguationSimilar Papers 제목 키워드 기반
The MUCOW word sense disambiguation test suite at WMT 2020
This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT systems to perform word sense disambiguation (WSD), i.…
NMTTranslationWord Sense DisambiguationAn Evaluation Benchmark for Testing the Word Sense Disambiguation Capabilities of Machine Translation Systems
Lexical ambiguity is one of the many challenging linguistic phenomena involved in translation, i.e., translating an ambiguous word with its correct sense. In this respect, previous work has shown that the translation qua…
Machine TranslationTranslationWord Sense DisambiguationNeural Machine Translation for Low Resource Languages using Bilingual Lexicon Induced from Comparable Corpora
Resources for the non-English languages are scarce and this paper addresses this problem in the context of machine translation, by automatically extracting parallel sentence pairs from the multilingual articles available…
ArticlesMachine TranslationNMTSentence+1SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
Evaluating software engineering capabilities has become a core component of modern large language models (LLMs); however, the key bottleneck hindering further scaling lies not in the scarcity of high-quality solutions, b…
Reinforcement LearningProgram RepairImproving Machine Translation of Rare and Unseen Word Senses
The performance of NMT systems has improved drastically in the past few years but the translation of multi-sense words still poses a challenge. Since word senses are not represented uniformly in the parallel corpora used…
Bilingual Lexicon InductionMachine TranslationNMTTranslation+2