CL-ReKD: Cross-lingual Knowledge Distillation for Multilingual Retrieval Question Answering
Cross-Lingual Retrieval Question Answering (CL-ReQA) is concerned with retrieving answer documents or passages to a question written in a different language. A common approach to CL-ReQA is to create a multilingual sentence embedding space such that question-answer pairs across different languages are close to each other. In this paper, we propose a novel CL-ReQA method utilizing the concept of knowledge distillation and a new cross-lingual consistency training technique to create a multilingual embedding space for ReQA. To assess the effectiveness of our work, we conducted comprehensive experiments on CL-ReQA and a downstream task, machine reading QA. We compared our proposed method with the current state-of-the-art solutions across three public CL-ReQA corpora. Our method outperforms competitors in 19 out of 21 settings of CL-ReQA. When used with a downstream machine reading QA task, our method outperforms the best existing language-model-based method by 10% in F1 while being 10 times faster in sentence embedding computation.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationLanguage ModelingLanguage ModellingQuestion AnsweringReading ComprehensionRetrievalSentenceSentence EmbeddingSentence-EmbeddingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation
Knowledge distillation (KD) has proven to be a successful strategy to improve the performance of smaller models in many NLP tasks. However, most of the work in KD only explores monolingual scenarios. In this paper, we in…
Knowledge DistillationDistilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferr…
Knowledge DistillationModel CompressionLanguage-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR
Modern LLM-based ASR systems have established multilingual capability as a standard feature, leveraging large-scale multilingual corpora and LLMs' cross-lingual knowledge to achieve competitive performance across multili…
Reinforcement LearningC2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval
Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for other languages lags behind English. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to …
Knowledge DistillationRetrievalVideo RetrievalVideo SimilarityLimitations of Knowledge Distillation for Zero-shot Transfer Learning
Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency …
CPUCross-Lingual TransferKnowledge DistillationTransfer Learning+1