A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
Knowledge distillation (KD) is a substantial strategy for transferring learned knowledge from one neural network model to another. A vast number of methods have been developed for this strategy. While most method designs a more efficient way to facilitate knowledge transfer, less attention has been put on comparing the effect of knowledge sources such as features, logits, and gradients. This work provides a new perspective to motivate a set of knowledge distillation strategies by approximating the classical KL-divergence criteria with different knowledge sources, making a systematic comparison possible in model compression and incremental learning. Our analysis indicates that logits are generally a more efficient knowledge source and suggests that having sufficient feature dimensions is crucial for the model design, providing a practical guideline for effective KD-based transfer learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Incremental LearningKnowledge DistillationModel CompressionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Closer Look at Knowledge Distillation in Spiking Neural Network Training
Spiking Neural Networks (SNNs) become popular due to excellent energy efficiency, yet facing challenges for effective model training. Recent works improve this by introducing knowledge distillation (KD) techniques, with …
Knowledge DistillationNormKD: Normalized Logits for Knowledge Distillation
Logit based knowledge distillation gets less attention in recent years since feature based methods perform better in most cases. Nevertheless, we find it still has untapped potential when we re-investigate the temperatur…
image-classificationImage ClassificationKnowledge DistillationHeterogeneous Complementary Distillation
Knowledge distillation (KD)transfers the dark knowledge from a complex teacher to a compact student. However, heterogeneous architecture distillation, such as Vision Transformer (ViT) to ResNet18, faces challenges due to…
Knowledge DistillationTopKD: Top-scaled Knowledge Distillation
Recent advances in knowledge distillation (KD) predominantly emphasize feature-level knowledge transfer, frequently overlooking critical information embedded within the teacher's logit distributions. In this paper, we re…
Knowledge DistillationDecoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
Knowledge Distillation (KD), a learning manner with a larger teacher network guiding a smaller student network, transfers dark knowledge from the teacher to the student via logits or intermediate features, with the aim o…
Knowledge DistillationPhilosophy