Understanding Self-Distillation and Partial Label Learning in Multi-Class Classification with Label Noise
Self-distillation (SD) is the process of training a student model using the outputs of a teacher model, with both models sharing the same architecture. Our study theoretically examines SD in multi-class classification with cross-entropy loss, exploring both multi-round SD and SD with refined teacher outputs, inspired by partial label learning (PLL). By deriving a closed-form solution for the student model's outputs, we discover that SD essentially functions as label averaging among instances with high feature correlations. Initially beneficial, this averaging helps the model focus on feature clusters correlated with a given instance for predicting the label. However, it leads to diminishing performance with increasing distillation rounds. Additionally, we demonstrate SD's effectiveness in label noise scenarios and identify the label corruption condition and minimum number of distillation rounds needed to achieve 100% classification accuracy. Our study also reveals that one-step distillation with refined teacher outputs surpasses the efficacy of multi-step SD using the teacher's direct output in high noise rate regimes.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-class ClassificationPartial Label LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning
Recent work on 4D point cloud sequences has attracted a lot of attention. However, obtaining exhaustively labeled 4D datasets is often very expensive and laborious, so it is especially important to investigate how to uti…
Knowledge DistillationRepresentation LearningPSScreen: Partially Supervised Multiple Retinal Disease Screening
Leveraging multiple partially labeled datasets to train a model for multiple retinal disease screening reduces the reliance on fully annotated datasets, but remains challenging due to significant domain shifts across tra…
Domain GeneralizationBayesian Optimization Meets Self-Distillation
Bayesian optimization (BO) has contributed greatly to improving model performance by suggesting promising hyperparameter configurations iteratively based on observations from multiple training trials. However, only parti…
Bayesian Optimizationimage-classificationImage ClassificationLearning with noisy labels+1Target-driven Self-Distillation for Partial Observed Trajectories Forecasting
Accurate prediction of future trajectories of traffic agents is essential for ensuring safe autonomous driving. However, partially observed trajectories can significantly degrade the performance of even state-of-the-art …
Autonomous DrivingKnowledge DistillationMotion ForecastingSelf-distillation for surgical action recognition
Surgical scene understanding is a key prerequisite for contextaware decision support in the operating room. While deep learning-based approaches have already reached or even surpassed human performance in various fields,…
Action RecognitionMedical Image AnalysisMulti-Task LearningScene Understanding