Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading
Automatic disease image grading is a significant application of artificial intelligence for healthcare, enabling faster and more accurate patient assessments. However, domain shifts, which are exacerbated by data imbalance, introduce bias into the model, posing deployment difficulties in clinical applications. To address the problem, we propose a novel \textbf{U}ncertainty-aware \textbf{M}ulti-experts \textbf{K}nowledge \textbf{D}istillation (UMKD) framework to transfer knowledge from multiple expert models to a single student model. Specifically, to extract discriminative features, UMKD decouples task-agnostic and task-specific features with shallow and compact feature alignment in the feature space. At the output space, an uncertainty-aware decoupled distillation (UDD) mechanism dynamically adjusts knowledge transfer weights based on expert model uncertainties, ensuring robust and reliable distillation. Additionally, UMKD also tackles the problems of model architecture heterogeneity and distribution discrepancies between source and target domains, which are inadequately tackled by previous KD approaches. Extensive experiments on histology prostate grading (\textit{SICAPv2}) and fundus image grading (\textit{APTOS}) demonstrate that UMKD achieves a new state-of-the-art in both source-imbalanced and target-imbalanced scenarios, offering a robust and practical solution for real-world disease image grading.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationTransfer LearningSimilar Papers 제목 키워드 기반
Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is chall…
Knowledge DistillationHow Is Uncertainty Propagated in Knowledge Distillation?
Knowledge distillation transfers behavior from a teacher to a student model, but the process is inherently stochastic: teacher outputs, student training, and student inference can all be random. Collapsing these uncertai…
Knowledge DistillationMulti-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
Knowledge distillation is a powerful method for model compression, enabling the efficient deployment of complex deep learning models (teachers), including large language models. However, its underlying statistical mechan…
Knowledge DistillationImage ClassificationBayesian InferenceModel CompressionComputation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an essential technique for model compression and transfer learning.…
InformativenessKnowledge DistillationModel CompressionTransfer LearningUncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Knowledge distillation has emerged as a powerful technique for model compression, enabling the transfer of knowledge from large teacher networks to compact student models. However, traditional knowledge distillation meth…
Knowledge DistillationImage ClassificationModel Compression