EA-KD: Entropy-based Adaptive Knowledge Distillation
Knowledge distillation (KD) enables a smaller "student" model to mimic a larger "teacher" model by transferring knowledge from the teacher's output or features. However, most KD methods treat all samples uniformly, overlooking the varying learning value of each sample and thereby limiting effectiveness. In this paper, we propose Entropy-based Adaptive Knowledge Distillation (EA-KD), a simple yet effective plug-and-play KD method that prioritizes learning from valuable samples. EA-KD quantifies each sample's learning value by strategically combining the entropy of the teacher and student output, then dynamically reweights the distillation loss to place greater emphasis on high-value samples. Extensive experiments across diverse KD frameworks and tasks$\unicode{x2014}$including image classification, object detection, and large language model (LLM) distillation$\unicode{x2014}$demonstrate that EA-KD consistently enhances performance, achieving state-of-the-art results with negligible computational cost. Our code will be publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationKnowledge DistillationLanguage ModelingLanguage ModellingLarge Language Modelobject-detectionObject DetectionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
Large language models (LLMs) have achieved remarkable performance across diverse domains, yet their enormous computational and memory requirements hinder deployment in resource-constrained environments. Knowledge distill…
Knowledge DistillationRethinking Reverse KL as Adaptive Entropy Distillation
Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In …
Mathematical ReasoningKnowledge DistillationDomain Adaptive Knowledge Distillation for Driving Scene Semantic Segmentation
Practical autonomous driving systems face two crucial challenges: memory constraints and domain gap issues. In this paper, we present a novel approach to learn domain adaptive knowledge in models with limited memory, thu…
Autonomous DrivingKnowledge DistillationSemantic SegmentationContext-Aware Knowledge Distillation with Adaptive Weighting for Image Classification
Knowledge distillation (KD) is a widely used technique to transfer knowledge from a large teacher network to a smaller student model. Traditional KD uses a fixed balancing factor alpha as a hyperparameter to combine the …
Knowledge DistillationImage ClassificationDynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing
The knowledge distillation uses a high-performance teacher network to guide the student network. However, the performance gap between the teacher and student networks can affect the student's training. This paper propose…
Knowledge Distillation