Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective
In this paper, we analyze the feature-based knowledge distillation for recommendation from the frequency perspective. By defining knowledge as different frequency components of the features, we theoretically demonstrate that regular feature-based knowledge distillation is equivalent to equally minimizing losses on all knowledge and further analyze how this equal loss weight allocation method leads to important knowledge being overlooked. In light of this, we propose to emphasize important knowledge by redistributing knowledge weights. Furthermore, we propose FreqD, a lightweight knowledge reweighting method, to avoid the computational cost of calculating losses on each knowledge. Extensive experiments demonstrate that FreqD consistently and significantly outperforms state-of-the-art knowledge distillation methods for recommender systems. Our code is available at https://github.com/woriazzc/KDs.
Code (1)
Tasks
Knowledge DistillationRecommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DE-RRD: A Knowledge Distillation Framework for Recommender System
Recent recommender systems have started to employ knowledge distillation, which is a model compression technique distilling knowledge from a cumbersome model (teacher) to a compact model (student), to reduce inference la…
Knowledge DistillationModel CompressionRecommendation SystemsRejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems
This paper analyzes Cross-Entropy (CE) loss in knowledge distillation (KD) for recommender systems. KD for recommender systems targets at distilling rankings, especially among items most likely to be preferred, and can o…
Knowledge DistillationKnowledge Distillation Approaches for Accurate and Efficient Recommender System
Despite its breakthrough in classification problems, Knowledge distillation (KD) to recommendation models and ranking problems has not been studied well in the previous literature. This dissertation is devoted to develop…
Knowledge DistillationRecommendation SystemsPreference-Consistent Knowledge Distillation for Recommender System
Feature-based knowledge distillation has been applied to compress modern recommendation models, usually with projectors that align student (small) recommendation models' dimensions with teacher dimensions. However, exist…
Knowledge DistillationRecommendation SystemsPromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning
Multimedia online platforms (e.g., Amazon, TikTok) have greatly benefited from the incorporation of multimedia (e.g., visual, textual, and acoustic) content into their personal recommender systems. These modalities provi…
Knowledge DistillationModel CompressionRecommendation Systems