Code-Switching with Word Senses for Pretraining in Neural Machine Translation
Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT), with many state-of-the-art (SOTA) NMT systems struggling to handle polysemous words (Campolungo et al., 2022). The same holds for the NMT pretraining paradigm of denoising synthetic "code-switched" text (Pan et al., 2021; Iyer et al., 2023), where word senses are ignored in the noising stage -- leading to harmful sense biases in the pretraining data that are subsequently inherited by the resulting models. In this work, we introduce Word Sense Pretraining for Neural Machine Translation (WSP-NMT) - an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. Our experiments show significant improvements in overall translation quality. Then, we show the robustness of our approach to scale to various challenging data and resource-scarce scenarios and, finally, report fine-grained accuracy improvements on the DiBiMT disambiguation benchmark. Our studies yield interesting and novel insights into the merits and challenges of integrating word sense information and structured knowledge in multilingual pretraining for NMT.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingMachine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology
An important question concerning contextualized word embedding (CWE) models like BERT is how well they can represent different word senses, especially those in the long tail of uncommon senses. Rather than build a WSD sy…
RetrievalWord Sense-Aware Machine Translation: Including Senses as Contextual Features for Improved Translation Models
Although it is commonly assumed that word sense disambiguation (WSD) should help to improve lexical choice and improve the quality of machine translation systems, how to successfully integrate word senses into such syste…
Machine TranslationTranslationWord Sense DisambiguationEncoders Help You Disambiguate Word Senses in Neural Machine Translation
Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explo…
DecoderMachine TranslationNMTTranslation+1Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders
A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We prop…
Word Sense DisambiguationMoving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders
A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We prop…
Word Sense Disambiguation