Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
We propose a modular architecture of language-specific encoder-decoders that constitutes a multilingual machine translation system that can be incrementally extended to new languages without the need for retraining the existing system when adding new languages. Differently from previous works, we simultaneously train $N$ languages in all translation directions by alternately freezing encoder or decoder modules, which indirectly forces the system to train in a common intermediate representation for all languages. Experimental results from multilingual machine translation show that we can successfully train this modular architecture improving on the initial languages while falling slightly behind when adding new languages or doing zero-shot translation. Additional comparison of the quality of sentence representation in the task of natural language inference shows that the alternately freezing training is also beneficial in this direction.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationNatural Language InferenceSentenceTranslationSimilar Papers 제목 키워드 기반
Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies
Transfer learning or multilingual model is essential for low-resource neural machine translation (NMT), but the applicability is limited to cognate languages by sharing their vocabularies. This paper shows effective tech…
Cross-Lingual TransferLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine Translation+3Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates …
DecoderMachine TranslationTranslationLightweight Adapter Tuning for Multilingual Speech Translation
Adapter modules were recently introduced as an efficient alternative to fine-tuning in NLP. Adapter tuning consists in freezing pretrained parameters of a model and injecting lightweight modules between layers, resulting…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+3Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings
In this work we look into adding a new language to a multilingual NMT system in an unsupervised fashion. Under the utilization of pre-trained cross-lingual word embeddings we seek to exploit a language independent multil…
Cross-Lingual Word EmbeddingsMachine TranslationNMTSentence+3Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training a…
Reading ComprehensionContinual PretrainingLanguage AcquisitionGeneral Knowledge