Understanding BERT Rankers Under Distillation
Deep language models such as BERT pre-trained on large corpus have given a huge performance boost to the state-of-the-art information retrieval ranking systems. Knowledge embedded in such models allows them to pick up complex matching signals between passages and queries. However, the high computation cost during inference limits their deployment in real-world search scenarios. In this paper, we study if and how the knowledge for search within BERT can be transferred to a smaller ranker through distillation. Our experiments demonstrate that it is crucial to use a proper distillation procedure, which produces up to nine times speedup while preserving the state-of-the-art performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving Bi-encoder Document Ranking Models with Two Rankers and Multi-teacher Distillation
BERT-based Neural Ranking Models (NRMs) can be classified according to how the query and document are encoded through BERT's self-attention layers - bi-encoder versus cross-encoder. Bi-encoder models are highly efficient…
Document RankingHow Different are Pre-trained Transformers for Text Ranking?
In recent years, large pre-trained transformers have led to substantial gains in performance over traditional retrieval models and feedback approaches. However, these results are primarily based on the MS Marco/TREC Deep…
Passage RetrievalRetrievalDistillation versus Contrastive Learning: How to Train Your Rerankers
Training effective text rerankers is crucial for information retrieval. Two strategies are widely used: contrastive learning (optimizing directly on ground-truth labels) and knowledge distillation (transferring knowledge…
Knowledge DistillationInformation RetrievalContrastive LearningOn the Calibration and Uncertainty of Neural Learning to Rank Models
According to the Probability Ranking Principle (PRP), ranking documents in decreasing order of their probability of relevance leads to an optimal document ranking for ad-hoc retrieval. The PRP holds when two conditions a…
Document RankingLearning-To-RankRetrievalTabRank: Chain-of-Thought Distillation for Table Re-Rankers
The ability to retrieve relevant tables for answering questions is a key task for structured information retrieval. Multi-stage retrieval systems rely heavily on rerankers to refine candidate lists produced by efficient …
Information RetrievalNatural QuestionsPassage RetrievalTable Retrieval