Improving Neural Ranking via Lossless Knowledge Distillation
We explore a novel perspective of knowledge distillation (KD) for learning to rank (LTR), and introduce Self-Distilled neural Rankers (SDR), where student rankers are parameterized identically to their teachers. Unlike the existing ranking distillation work which pursues a good trade-off between performance and efficiency, SDR is able to significantly improve ranking performance of students over the teacher rankers without increasing model capacity. The key success factors of SDR, which differs from common distillation techniques for classification are: (1) an appropriate teacher score transformation function, and (2) a novel listwise distillation framework. Both techniques are specifically designed for ranking problems and are rarely studied in the existing knowledge distillation literature. Building upon the state-of-the-art neural ranking structure, SDR is able to push the limits of neural ranking performance above a recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics, the first time in the literature. In addition to the strong empirical results, we give theoretical explanations on why listwise distillation is effective for neural rankers, and provide ablation studies to verify the necessity of the key factors in the SDR framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationLearning-To-RankMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RankDistil: Knowledge Distillation for Ranking
Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher.Despite its success in several deep learning applications, the study of distillation is most…
Document RankingKnowledge DistillationBoosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation
Lossless speculative decoding accelerates target large language model (LLM) inference by employing a lightweight draft model for generating tree-structured candidates, which are subsequently verified in parallel by the t…
Knowledge DistillationLanguage ModellingLarge Language ModelMathematical Reasoning+2Knowledge Distillation Approaches for Accurate and Efficient Recommender System
Despite its breakthrough in classification problems, Knowledge distillation (KD) to recommendation models and ranking problems has not been studied well in the previous literature. This dissertation is devoted to develop…
Knowledge DistillationRecommendation SystemsAn Empirical Study of Uniform-Architecture Knowledge Distillation in Document Ranking
Although BERT-based ranking models have been commonly used in commercial search engines, they are usually time-consuming for online ranking tasks. Knowledge distillation, which aims at learning a smaller model with compa…
Document RankingKnowledge DistillationImproving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
Retrieval and ranking models are the backbone of many applications such as web search, open domain QA, or text-based recommender systems. The latency of neural ranking models at query time is largely dependent on the arc…
Knowledge DistillationPassage RankingRecommendation SystemsRe-Ranking+1