Dual Correction Strategy for Ranking Distillation in Top-N Recommender System
Knowledge Distillation (KD), which transfers the knowledge of a well-trained large model (teacher) to a small model (student), has become an important area of research for practical deployment of recommender systems. Recently, Relaxed Ranking Distillation (RRD) has shown that distilling the ranking information in the recommendation list significantly improves the performance. However, the method still has limitations in that 1) it does not fully utilize the prediction errors of the student model, which makes the training not fully efficient, and 2) it only distills the user-side ranking information, which provides an insufficient view under the sparse implicit feedback. This paper presents Dual Correction strategy for Distillation (DCD), which transfers the ranking information from the teacher model to the student model in a more efficient manner. Most importantly, DCD uses the discrepancy between the teacher model and the student model predictions to decide which knowledge to be distilled. By doing so, DCD essentially provides the learning guidance tailored to "correcting" what the student model has failed to accurately predict. This process is applied for transferring the ranking information from the user-side as well as the item-side to address sparse implicit user feedback. Our experiments show that the proposed method outperforms the state-of-the-art baselines, and ablation studies validate the effectiveness of each component.
Code (1)
Tasks
Knowledge DistillationRecommendation SystemsSimilar Papers 제목 키워드 기반
Knowledge Distillation Approaches for Accurate and Efficient Recommender System
Despite its breakthrough in classification problems, Knowledge distillation (KD) to recommendation models and ranking problems has not been studied well in the previous literature. This dissertation is devoted to develop…
Knowledge DistillationRecommendation SystemsDE-RRD: A Knowledge Distillation Framework for Recommender System
Recent recommender systems have started to employ knowledge distillation, which is a model compression technique distilling knowledge from a cumbersome model (teacher) to a compact model (student), to reduce inference la…
Knowledge DistillationModel CompressionRecommendation SystemsRejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems
This paper analyzes Cross-Entropy (CE) loss in knowledge distillation (KD) for recommender systems. KD for recommender systems targets at distilling rankings, especially among items most likely to be preferred, and can o…
Knowledge DistillationCross-Task Knowledge Distillation in Multi-Task Recommendation
Multi-task learning (MTL) has been widely used in recommender systems, wherein predicting each type of user feedback on items (e.g, click, purchase) are treated as individual tasks and jointly trained with a unified mode…
Knowledge DistillationMulti-Task LearningRecommendation SystemsTransfer LearningRanking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectivenes…
Knowledge DistillationLearning-To-RankRecommendation SystemsVocal Bursts Intensity Prediction