The Role of Teacher Calibration in Knowledge Distillation
Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student's performance. In this paper, we reveal a strong correlation between the teacher's calibration error and the student's accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher's calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationModel CompressionSimilar Papers 제목 키워드 기반
LoCa: Logit Calibration for Knowledge Distillation
Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the output logits. However, we find a common i…
image-classificationImage ClassificationKnowledge DistillationModel Compression+1Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model
In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that hol…
Knowledge DistillationLanguage ModelingLanguage ModellingSentence+1Rethinking the Knowledge Distillation From the Perspective of Model Calibration
Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous…
Knowledge DistillationFaithful Knowledge Distillation
Knowledge distillation (KD) has received much attention due to its success in compressing networks to allow for their deployment in resource-constrained systems. While the problem of adversarial robustness has been studi…
Adversarial RobustnessKnowledge DistillationDistillation Traps and Guards: A Calibration Knob for LLM Distillability
Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks. Our analysis revealed several distillation t…
Knowledge Distillation