Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 {\textperthousand} of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.
Code (1)
Tasks
Contrastive LearningIn-Context LearningSentenceTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many commun…
A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing…
Cross-Lingual TransferTranslationWord EmbeddingsMultilingual Vision-Language Models, A Survey
This survey examines multilingual vision-language models that process text and images across languages. We review 33 models and 23 benchmarks, spanning encoder-only and generative architectures, and identify a key tensio…
Contrastive LearningMultilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction
We present a study on leveraging multilingual pre-trained generative language models for zero-shot cross-lingual event argument extraction (EAE). By formulating EAE as a language generation task, our method effectively e…
Cross-Lingual TransferEvent Argument ExtractionText GenerationZero-Shot Cross-Lingual TransferMultilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction
We present a pioneering study on leveraging multilingual pre-trained generative language models for zero-shot cross-lingual event argument extraction (EAE) by formulating EAE as a language generation task. Compared to pr…
Cross-Lingual TransferEvent Argument ExtractionText Generation