Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies
Transfer learning or multilingual model is essential for low-resource neural machine translation (NMT), but the applicability is limited to cognate languages by sharing their vocabularies. This paper shows effective techniques to transfer a pre-trained NMT model to a new, unrelated language without shared vocabularies. We relieve the vocabulary mismatch by using cross-lingual word embedding, train a more language-agnostic encoder by injecting artificial noises, and generate synthetic data easily from the pre-training data without back-translation. Our methods do not require restructuring the vocabulary or retraining the model. We improve plain NMT transfer by up to +5.1% BLEU in five low-resource translation tasks, outperforming multilingual joint training by a large margin. We also provide extensive ablation studies on pre-trained embedding, synthetic data, vocabulary size, and parameter freezing for a better understanding of NMT transfer.
Code (1)
Tasks
Cross-Lingual TransferLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine TranslationNMTTransfer LearningTranslationSimilar Papers 제목 키워드 기반
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model. Its improved translation perf…
Cross-Lingual TransferMachine TranslationNMTTransfer Learning+1Analyzing and Improving Cross-lingual Knowledge Transfer for Machine Translation
Multilingual machine translation systems aim to make knowledge accessible across languages, yet learning effective cross-lingual representations remains challenging. These challenges are especially pronounced for low-res…
Machine TranslationCross-lingual Word Embeddings beyond Zero-shot Machine Translation
We explore the transferability of a multilingual neural machine translation model to unseen languages when the transfer is grounded solely on the cross-lingual word embeddings. Our experimental results show that the tran…
Cross-Lingual Word EmbeddingsMachine TranslationTranslationWord Embeddings+1Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
Transfer learning between different language pairs has shown its effectiveness for Neural Machine Translation (NMT) in low-resource scenario. However, existing transfer methods involving a common target language are far …
Machine TranslationNMTTransfer LearningTranslationCan Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?
Multilingual pretraining and fine-tuning have remarkably succeeded in various natural language processing tasks. Transferring representations from one language to another is especially crucial for cross-lingual learning.…
Cross-Lingual TransferMachine TranslationNatural Language UnderstandingRepresentation Learning+3