Multilingual Representation Distillation with Contrastive Learning
Multilingual sentence representations from large models encode semantic information from two or more languages and can be used for different cross-lingual information retrieval and matching tasks. In this paper, we integrate contrastive learning into multilingual representation distillation and use it for quality estimation of parallel sentences (i.e., find semantically similar sentences that can be used as translations of each other). We validate our approach with multilingual similarity search and corpus filtering tasks. Experiments across different low-resource languages show that our method greatly outperforms previous sentence encoders such as LASER, LASER3, and LaBSE.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningCross-Lingual Information RetrievalInformation RetrievalRetrievalSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new…
Contrastive LearningKnowledge DistillationModel Compressionspeech-recognition+4mCLIP: Multilingual CLIP via Cross-lingual Transfer
Large-scale vision-language pretrained (VLP) models like CLIP have shown remarkable performance on various downstream cross-modal tasks. However, they are usually biased towards English due to the lack of sufficient non-…
Contrastive LearningCross-Lingual TransferCross-Modal RetrievalImage-text Retrieval+7MIMO: Multilingual Information Retrieval via Monolingual Objectives
Multilingual Information Retrieval (MLIR) reflects real-world search environments in which queries and relevant documents may appear in different languages within a mixed-language corpus. However, existing embedding mode…
Knowledge DistillationInformation RetrievalContrastive LearningPrototypical Contrastive Predictive Coding
Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model to a smaller s…
Contrastive LearningKnowledge DistillationModel CompressionSelf-Supervised LearningProbing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, i…
Contrastive LearningCross-Lingual Entity LinkingEntity LinkingPretrained Multilingual Language Models+4