Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning
Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very easy to create without requiring multilingual expertise, so this offers a more efficient alternative for low-resource languages. We find that multilingual image-caption alignment can implicitly align the text representations between languages, languages unseen by the encoder in pretraining can be incorporated into this alignment post-hoc, and these aligned representations are usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval.
Code (1)
Tasks
Natural Language UnderstandingRetrievalSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lin…
Contrastive LearningSemantic SimilarityImproving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages a…
Contrastive LearningIn-Context LearningSentenceTransfer LearningSupervised Contrastive Learning for Cross-lingual Transfer Learning
“Multilingual pre-trained representations are not well-aligned by nature, which harms their performance on cross-lingual tasks. Previous methods propose to post-align the multilingual pretrained representations by multi-…
Contrastive LearningCross-Lingual TransferTransfer LearningImproving Multi-lingual Alignment Through Soft Contrastive Learning
Making decent multi-lingual sentence representations is critical to achieve high performances in cross-lingual downstream tasks. In this work, we propose a novel method to align multi-lingual embeddings based on the simi…
Contrastive LearningSentenceSTSHC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding
State-of-the-art model for zero-shot cross-lingual spoken language understanding performs cross-lingual unsupervised contrastive learning to achieve the label-agnostic semantic alignment between each utterance and its co…
Contrastive LearningSpoken Language Understanding