Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation
Successful methods for unsupervised neural machine translation (UNMT) employ crosslingual pretraining via self-supervision, often in the form of a masked language modeling or a sequence generation task, which requires the model to align the lexical- and high-level representations of the two languages. While cross-lingual pretraining works for similar languages with abundant corpora, it performs poorly in low-resource and distant languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings. Empirical results demonstrate improved performance both on UNMT (up to 4.5 BLEU) and bilingual lexicon induction using our method compared to a UNMT baseline.
Code (1)
Tasks
Bilingual Lexicon InductionLanguage ModelingLanguage ModellingMachine TranslationMasked Language ModelingTranslationSimilar Papers 제목 키워드 기반
MANTIS at TSAR-2022 Shared Task: Improved Unsupervised Lexical Simplification with Pretrained Encoders
In this paper we present our contribution to the TSAR-2022 Shared Task on Lexical Simplification of the EMNLP 2022 Workshop on Text Simplification, Accessibility, and Readability. Our approach builds on and extends the u…
Language ModelingLanguage ModellingLexical SimplificationNatural Language Inference+3The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translation
Unsupervised on-the-fly back-translation, in conjunction with multilingual pretraining, is the dominant method for unsupervised neural machine translation. Theoretically, however, the method should not work in general. W…
Machine TranslationTranslationUnsupervised Machine TranslationIncremental Neural Lexical Coherence Modeling
Pretrained language models, neural models pretrained on massive amounts of data, have established the state of the art in a range of NLP tasks. They are based on a modern machine-learning technique, the Transformer which…
Language ModelingLanguage ModellingMultilingual Lexical Simplification via Paraphrase Generation
Lexical simplification (LS) methods based on pretrained language models have made remarkable progress, generating potential substitutes for a complex word through analysis of its contextual surroundings. However, these m…
DiversityLexical SimplificationMachine TranslationParaphrase Generation+2Iterative Constrained Back-Translation for Unsupervised Domain Adaptation of Machine Translation
Back-translation has been proven to be effective in unsupervised domain adaptation of neural machine translation (NMT). However, the existing back-translation methods mainly improve domain adaptability by generating in-d…
Domain AdaptationMachine TranslationNMTSentence+3