Regularizing Class-wise Predictions via Self-knowledge Distillation
Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.
Code (1)
Tasks
image-classificationImage ClassificationKnowledge DistillationSelf-Knowledge DistillationSimilar Papers 제목 키워드 기반
Multi-Body Neural Scene Flow
The test-time optimization of scene flow - using a coordinate network as a neural prior - has gained popularity due to its simplicity, lack of dataset bias, and state-of-the-art performance. We observe, however, that alt…
Scene Flow EstimationTrajectory PredictionRationalizing Transformer Predictions via End-To-End Differentiable Self-Training
We propose an end-to-end differentiable training paradigm for stable training of a rationalized transformer classifier. Our approach results in a single model that simultaneously classifies a sample and scores input toke…
Neighbor Class Consistency on Unsupervised Domain Adaptation
Unsupervised domain adaptation (UDA) is to make predictions for unlabeled data in a target domain with labeled data from source domain available. Recent advances exploit entropy minimization and self-training to align th…
ClusteringDomain Adaptationimage-classificationImage Classification+1Few-Shot Class-Incremental Learning via Class-Aware Bilateral Distillation
Few-Shot Class-Incremental Learning (FSCIL) aims to continually learn novel classes based on only few training samples, which poses a more challenging task than the well-studied Class-Incremental Learning (CIL) due t…
class-incremental learningClass Incremental LearningFew-Shot Class-Incremental LearningGeneral Knowledge+3Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving
In autonomous systems, precise object detection and uncertainty estimation are critical for self-aware and safe operation. This work addresses confidence calibration for the classification task of 3D object detectors. We…
Autonomous DrivingObject Detection