How Train-Test Leakage Affects Zero-shot Retrieval
Neural retrieval models are often trained on (subsets of) the millions of queries of the MS MARCO / ORCAS datasets and then tested on the 250 Robust04 queries or other TREC benchmarks with often only 50 queries. In such setups, many of the few test queries can be very similar to queries from the huge training data -- in fact, 69% of the Robust04 queries have near-duplicates in MS MARCO / ORCAS. We investigate the impact of this unintended train-test leakage by training neural retrieval models on combinations of a fixed number of MS MARCO / ORCAS queries that are highly similar to the actual test queries and an increasing number of other queries. We find that leakage can improve effectiveness and even change the ranking of systems. However, these effects diminish as the amount of leakage among all training instances decreases and thus becomes more realistic.
Code (1)
Tasks
RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revisiting the Effects of Leakage on Dependency Parsing
Recent work by S{\o}gaard (2020) showed that, treebank size aside, overlap between training and test graphs (termed leakage) explains more of the observed variation in dependency parsing performance than other explanatio…
Dependency ParsingMemory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks
Generative Large Language Models (LLMs) have shown promising results in text annotation using zero-shot and few-shot learning. Yet these approaches do not allow the model to retain information from previous annotations, …
AllFew-Shot Learningtext annotationQ-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment
We observe that zero-shot appearance transfer with large-scale image generation models faces a significant challenge: Attention Leakage. This challenge arises when the semantic mapping between two images is captured by t…
Image GenerationSubword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation
Zero-shot neural machine translation is an attractive goal because of the high cost of obtaining data and building translation systems for new translation directions. However, previous papers have reported mixed success …
Machine TranslationSegmentationTAGTranslationGenerative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement
This paper presents a new challenge that calls for zero-shot text-to-speech (TTS) systems to augment speech data for the downstream task, personalized speech enhancement (PSE), as part of the Generative Data Augmentation…
Data AugmentationSpeech EnhancementSpeech SynthesisSynthetic Data Generation+2