Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Knowledge distillation has emerged as a powerful technique for model compression, enabling the transfer of knowledge from large teacher networks to compact student models. However, traditional knowledge distillation methods treat all teacher predictions equally, regardless of the teacher's confidence in those predictions. This paper proposes an uncertainty-aware dual-student knowledge distillation framework that leverages teacher prediction uncertainty to selectively guide student learning. We introduce a peer-learning mechanism where two heterogeneous student architectures, specifically ResNet-18 and MobileNetV2, learn collaboratively from both the teacher network and each other. Experimental results on ImageNet-100 demonstrate that our approach achieves superior performance compared to baseline knowledge distillation methods, with ResNet-18 achieving 83.84\% top-1 accuracy and MobileNetV2 achieving 81.46\% top-1 accuracy, representing improvements of 2.04\% and 0.92\% respectively over traditional single-student distillation approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationImage ClassificationModel CompressionResults from the Paper
| Rank | Task | Dataset | Model | Metrics |
|---|---|---|---|---|
| #391 | Image Classification | ImageNet | Uncertainty-Aware Dual-Student Knowledge | Top 1 Accuracy: 83.84 |
| #5 | Knowledge Distillation | ImageNet | Uncertainty-Aware Dual-Student Knowledge | Top-1 accuracy %: 83.84 |
| #1 | Model Compression | ImageNet | Uncertainty-Aware Dual-Student Knowledge | Top-1: 83.84 |
Similar Papers 제목 키워드 기반
How Is Uncertainty Propagated in Knowledge Distillation?
Knowledge distillation transfers behavior from a teacher to a student model, but the process is inherently stochastic: teacher outputs, student training, and student inference can all be random. Collapsing these uncertai…
Knowledge DistillationUncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is chall…
Knowledge DistillationComputation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an essential technique for model compression and transfer learning.…
InformativenessKnowledge DistillationModel CompressionTransfer LearningUncertainty-Aware Knowledge Distillation for Compact and Efficient 6DoF Pose Estimation
Compact and efficient 6DoF object pose estimation is crucial in applications such as robotics, augmented reality, and space autonomous navigation systems, where lightweight models are critical for real-time accurate perf…
Autonomous NavigationKnowledge DistillationPose EstimationSpacecraft Pose Estimation+1U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening
Conventional methods for PAN-sharpening often struggle to restore fine details due to limitations in leveraging high-frequency information. Moreover, diffusion-based approaches lack sufficient conditioning to fully utili…
Knowledge Distillation