XITE: Cross-lingual Interpolation for Transfer using Embeddings
Facilitating cross-lingual transfer in multilingual language models remains a critical challenge. Towards this goal, we propose an embedding-based data augmentation technique called XITE. We start with unlabeled text from a low-resource target language, identify an English counterpart in a task-specific training corpus using embedding-based similarities and adopt its label. Next, we perform a simple interpolation of the source and target embeddings to create synthetic data for task-specific fine-tuning. Projecting the target text into a language-rich subspace using linear discriminant analysis (LDA), prior to interpolation, further boosts performance. Our cross-lingual embedding-based augmentation technique XITE yields significant improvements of up to 35.91% for sentiment analysis and up to 81.16% for natural language inference, using XLM-R, for a diverse set of target languages including Korean, Arabic, Urdu and Hindi. Apart from boosting cross-lingual transfer, adaptation using XITE also safeguards against forgetting and maintains task performance on the high-resource language.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceCross-Lingual TransferSentiment AnalysisData AugmentationSimilar Papers 제목 키워드 기반
Zero-shot Cross-lingual Transfer is Under-specified Optimization
Pretrained multilingual encoders enable zero-shot cross-lingual transfer performance, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that high variance res…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferOn the Cross-lingual Transferability of Contextualized Sense Embeddings
In this paper we analyze the extent to which contextualized sense embeddings, i.e., sense embeddings that are computed based on contextualized word embeddings, are transferable across languages.To this end, we compiled a…
Word EmbeddingsWord Sense DisambiguationGender Bias in Multilingual Embeddings and Cross-Lingual Transfer
Multilingual representations embed words from many languages into a single semantic space such that words with similar meanings are close to each other regardless of the language. These embeddings have been widely used i…
Cross-Lingual TransferTransfer LearningCross-lingual Word Embeddings beyond Zero-shot Machine Translation
We explore the transferability of a multilingual neural machine translation model to unseen languages when the transfer is grounded solely on the cross-lingual word embeddings. Our experimental results show that the tran…
Cross-Lingual Word EmbeddingsMachine TranslationTranslationWord Embeddings+1Cross-lingual Transfer of Sentiment Classifiers
Word embeddings represent words in a numeric space so that semantic relations between words are represented as distances and directions in the vector space. Cross-lingual word embeddings transform vector spaces of differ…
BIG-bench Machine LearningCross-Lingual TransferCross-Lingual Word EmbeddingsWord Embeddings