Self-Knowledge Distillation via Dropout
To boost the performance, deep neural networks require deeper or wider network structures that involve massive computational and memory costs. To alleviate this issue, the self-knowledge distillation method regularizes the model by distilling the internal knowledge of the model itself. Conventional self-knowledge distillation methods require additional trainable parameters or are dependent on the data. In this paper, we propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout distills the posterior distributions of multiple models through a dropout sampling. Our method does not require any additional trainable modules, does not rely on data, and requires only simple operations. Furthermore, this simple method can be easily combined with various self-knowledge distillation approaches. We provide a theoretical and experimental analysis of the effect of forward and reverse KL-divergences in our work. Extensive experiments on various vision tasks, i.e., image classification, object detection, and distribution shift, demonstrate that the proposed method can effectively improve the generalization of a single network. Further experiments show that the proposed method also improves calibration performance, adversarial robustness, and out-of-distribution detection ability.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial Robustnessimage-classificationImage ClassificationKnowledge Distillationobject-detectionObject DetectionOut-of-Distribution DetectionSelf-Knowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
The accurate segmentation of brain tumors from multi-modal MRI is critical for clinical diagnosis and treatment planning. While integrating complementary information from various MRI sequences is a common practice, the f…
Brain Tumor SegmentationGNN's Uncertainty Quantification using Self-Distillation
Graph Neural Networks (GNNs) have shown remarkable performance in the healthcare domain. However, what remained challenging is quantifying the predictive uncertainty of GNNs, which is an important aspect of trustworthine…
Knowledge DistillationUncertainty QuantificationLayer-wise Regularized Dropout for Neural Language Models
Among the various pre-trained neural language models that are popular today, dropout is already an indispensable regularization technique. To solve the inconsistency between training and inference caused by the randomnes…
Abstractive Text SummarizationMachine TranslationNatural Language UnderstandingSelf-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
Few-shot learning aims to classify unseen classes with a few training examples. While recent works have shown that standard mini-batch training with a carefully designed training strategy can improve generalization abili…
Data AugmentationFew-Shot LearningMeta-Learning with Self-Improving Momentum Target
The idea of using a separately trained target model (or teacher) to improve the performance of the student model has been increasingly popular in various machine learning domains, and meta-learning is no exception; a rec…
Knowledge DistillationMeta-LearningMeta Reinforcement Learning