Knowledge Distillation from A Stronger Teacher

2022-05-21 · Tao Huang, Shan You, Fei Wang, Chen Qian, Chang Xu

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD .

📄 PDF Abstract BibTeX arXiv:2205.10536

Code (3)

hunto/dist_kd 공식 구현 pytorch

hunto/image_classification_sota pytorch

yoshitomo-matsubara/torchdistill pytorch

Tasks

image-classification Image Classification Knowledge Distillation Object Detection Semantic Segmentation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Strong Teacher Not Needed? On Distillation in LLM Pretraining

2026-05-22 · Taiming Lu, Zhuang Liu arxiv

Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By va…

Knowledge Distillation

Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching

2024-10-09 · Wenqi Niu, Yingchao Wang, Guohui Cai, Hanpo Hou

Knowledge Distillation (KD) has emerged as a pivotal technique for neural network compression and performance enhancement. Most KD methods aim to transfer dark knowledge from a cumbersome teacher model to a lightweight s…

Knowledge Distillation Neural Network Compression

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation

2023-05-16 · Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu 외

It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledg…

Knowledge Distillation text-classification Text Classification Transfer Learning

Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

2026-02-01 · Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu 외 arxiv

Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challen…

Knowledge Distillation

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

2026-09-01 · Jacqueline He, Howard Yen, Shuyue Stella Li, Margaret Li 외 hf

Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across training stages remains unclear. Through contr…

Self-Supervised Learning Knowledge Distillation