Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters
A popular approach to creating a zero-shot cross-language retrieval model is to substitute a monolingual pretrained language model in the retrieval model with a multilingual pretrained language model such as Multilingual BERT. This multilingual model is fined-tuned to the retrieval task with monolingual data such as English MS MARCO using the same training recipe as the monolingual retrieval model used. However, such transferred models suffer from mismatches in the languages of the input text during training and inference. In this work, we propose transferring monolingual retrieval models using adapters, a parameter-efficient component for a transformer network. By adding adapters pretrained on language tasks for a specific language with task-specific adapters, prior work has shown that the adapter-enhanced models perform better than fine-tuning the entire model when transferring across languages in various NLP tasks. By constructing dense retrieval models with adapters, we show that models trained with monolingual data are more effective than fine-tuning the entire model when transferring to a Cross Language Information Retrieval (CLIR) setting. However, we found that the prior suggestion of replacing the language adapters to match the target language at inference time is suboptimal for dense retrieval models. We provide an in-depth analysis of this discrepancy between other cross-language NLP tasks and CLIR.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalLanguage ModelingLanguage ModellingRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-s…
Cross-Lingual TransferNERPOSPOS Tagging+1Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation
With the release of new large language models (LLMs) like Llama and Mistral, zero-shot cross-lingual transfer has become increasingly feasible due to their multilingual pretraining and strong generalization capabilities.…
Zero-Shot Cross-Lingual Transferparameter-efficient fine-tuningOrthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Adapter modules have recently been used for efficient fine-tuning and language specialization of massively multilingual Transformers (MMTs), improving downstream zero-shot cross-lingual transfer. In this work, we propose…
Cross-Lingual TransferZero-Shot Cross-Lingual TransferImproving Cross-lingual Text Classification with Zero-shot Instance-Weighting
Cross-lingual text classification (CLTC) is a challenging task made even harder still due to the lack of labeled data in low-resource languages. In this paper, we propose zero-shot instance-weighting, a general model-agn…
Classificationtext-classificationText ClassificationZero-Shot LearningRealistic Zero-Shot Cross-Lingual Transfer in Legal Topic Classification
We consider zero-shot cross-lingual transfer in legal topic classification using the recent MultiEURLEX dataset. Since the original dataset contains parallel documents, which is unrealistic for zero-shot cross-lingual tr…
Cross-Lingual TransferTopic ClassificationTranslationZero-Shot Cross-Lingual Transfer