Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
While impressive performance has been achieved on the task of Answer Sentence Selection (AS2) for English, the same does not hold for languages that lack large labeled datasets. In this work, we propose Cross-Lingual Knowledge Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2 models for low-resource languages in the tasks without the need of labeled data for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a multilingual AS2 dataset with over 70K questions spanning 8 typologically diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2 with multiple teachers, diverse monolingual and multilingual pretrained language models (PLMs) as students, and both monolingual and multilingual training. The results demonstrate that CLKD either outperforms or rivals even supervised fine-tuning with the same amount of labeled data and a combination of machine translation and the teacher model. Our method can potentially enable stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the largest multilingual AS2 dataset for further studies in the research community.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine TranslationSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CL-ReKD: Cross-lingual Knowledge Distillation for Multilingual Retrieval Question Answering
Cross-Lingual Retrieval Question Answering (CL-ReQA) is concerned with retrieving answer documents or passages to a question written in a different language. A common approach to CL-ReQA is to create a multilingual sente…
Knowledge DistillationLanguage ModelingLanguage ModellingQuestion Answering+5cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillation
Vision-and-language tasks are gaining popularity in the research community, but the focus is still mainly on English. We propose a pipeline that utilizes English-only vision-language models to train a monolingual model f…
Knowledge DistillationQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)XD: Cross-lingual Knowledge Distillation for Polyglot Sentence Embeddings
Current state-of-the-art results in multilingual natural language inference (NLI) are based on tuning XLM (a pre-trained polyglot language model) separately for each language involved, resulting in multiple models. We re…
Knowledge DistillationLanguage ModelingLanguage ModellingNatural Language Inference+2Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, i…
Contrastive LearningCross-Lingual Entity LinkingEntity LinkingPretrained Multilingual Language Models+4Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation
Despite substantial progress in multilingual extractive Question Answering (QA), models with high and uniformly distributed performance across languages remain challenging, especially for languages with limited resources…
Cross-Lingual Question AnsweringCross-Lingual TransferExtractive Question-AnsweringKnowledge Distillation+3