Lexicon-Enhanced Self-Supervised Training for Multilingual Dense Retrieval
Recent multilingual pre-trained models have shown better performance in various multilingual tasks. However, these models perform poorly on multilingual retrieval tasks due to lacking multilingual training data. In this paper, we propose to mine and generate self-supervised training data based on a large-scale unlabeled corpus. We carefully design a mining method which combines the sparse and dense models to mine the relevance of unlabeled queries and passages. And we introduce a query generator to generate more queries in target languages for unlabeled passages. Through extensive experiments on Mr. TYDI dataset and an industrial dataset from a commercial search engine, we demonstrate that our method performs better than baselines based on various pre-trained multilingual models. Our method even achieves on-par performance with the supervised method on the latter dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSimilar Papers 제목 키워드 기반
Entity Enhanced BERT Pre-training for Chinese NER
Character-level BERT pre-trained in Chinese suffers a limitation of lacking lexicon information, which shows effectiveness for Chinese NER. To integrate the lexicon into pre-trained LMs for Chinese NER, we investigate a …
NERATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems
In this paper, a multilingual end-to-end framework, called as ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the propos…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Feature EngineeringRepresentation Learning+3Learning Unsupervised Multilingual Word Embeddings with Incremental Multilingual Hubs
Recent research has discovered that a shared bilingual word embedding space can be induced by projecting monolingual word embedding spaces from two languages using a self-learning paradigm without any bilingual supervisi…
Bilingual Lexicon InductionCross-Lingual Word EmbeddingsDependency ParsingDocument Classification+3Measuring South Asian Biases in Large Language Models
Evaluations of Large Language Models (LLMs) often overlook intersectional and culturally specific biases, particularly in underrepresented multilingual regions like South Asia. This work addresses these gaps by conductin…
Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer
Unsupervised cross-lingual transfer involves transferring knowledge between languages without explicit supervision. Although numerous studies have been conducted to improve performance in such tasks by focusing on cross-…
Cross-Lingual TransferGraph Attentionnamed-entity-recognitionNamed Entity Recognition+6