Towards Robust Neural Retrieval Models with Synthetic Pre-Training
Recent work has shown that commonly available machine reading comprehension (MRC) datasets can be used to train high-performance neural information retrieval (IR) systems. However, the evaluation of neural IR has so far been limited to standard supervised learning settings, where they have outperformed traditional term matching baselines. We conduct in-domain and out-of-domain evaluations of neural IR, and seek to improve its robustness across different scenarios, including zero-shot settings. We show that synthetic training examples generated using a sequence-to-sequence generator can be effective towards this goal: in our experiments, pre-training with synthetic examples improves retrieval performance in both in-domain and out-of-domain evaluation on five different test sets.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalMachine Reading ComprehensionReading ComprehensionRetrievalSimilar Papers 제목 키워드 기반
Understanding Synthetic Context Extension via Retrieval Heads
Long-context LLMs are increasingly in demand for applications such as retrieval-augmented generation. To defray the cost of pretraining LLMs over long contexts, recent work takes an approach of synthetic context extensio…
RetrievalRetrieval-augmented GenerationSynthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models
We address the challenge of ensuring differential privacy (DP) guarantees in training deep retrieval systems. Training these systems often involves the use of contrastive-style losses, which are typically non-per-example…
Privacy PreservingRecommendation SystemsRetrievalLeveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval
There has been limited success for dense retrieval models in multilingual retrieval, due to uneven and scarce training data available across multiple languages. Synthetic training data generation is promising (e.g., InPa…
Language ModelingLanguage ModellingLarge Language ModelRetrievalMulti-stage Training with Improved Negative Contrast for Neural Passage Retrieval
In the context of neural passage retrieval, we study three promising techniques: synthetic data generation, negative sampling, and fusion. We systematically investigate how these techniques contribute to the performance …
Passage RetrievalRetrievalSynthetic Data GenerationImproving Vietnamese Legal Document Retrieval using Synthetic Data
In the field of legal information retrieval, effective embedding-based models are essential for accurate question-answering systems. However, the scarcity of large annotated datasets poses a significant challenge, partic…
Information RetrievalQuestion AnsweringRetrieval