On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.
Code (1)
Tasks
Meta-LearningSimilar Papers 제목 키워드 기반
Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation
Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. Language-specific modeling methods show promise in reducing interference. However, they often rely on heu…
Cross-Lingual TransferMachine TranslationTransfer LearningTranslationHLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation
Multilingual neural machine translation (MNMT) trained in multiple language pairs has attracted considerable attention due to fewer model parameters and lower training costs by sharing knowledge among multiple languages.…
DecoderMachine TranslationTranslationMore Parameters? No Thanks!
This work studies the long-standing problems of model capacity and negative interference in multilingual neural machine translation MNMT. We use network pruning techniques and observe that pruning 50-70% of the parameter…
Learning Language specific modelsMachine TranslationTranslationAn Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models
The capacity and effectiveness of pre-trained multilingual models (MLMs) for zero-shot cross-lingual transfer is well established. However, phenomena of positive or negative transfer, and the effect of language choice st…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferAdaptive Sparse Transformer for Multilingual Translation
Multilingual machine translation has attracted much attention recently due to its support of knowledge transfer among languages and the low cost of training and deployment compared with numerous bilingual models. A known…
Machine TranslationTransfer LearningTranslation