What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
Code (1)
Tasks
Active LearningData AugmentationKnowledge DistillationSimilar Papers 제목 키워드 기반
What Makes Better Augmentation Strategies? Augment Difficult but Not too Different
The practice of data augmentation has been extensively used to boost the performance of deep neural networks for various NLP tasks. It is more effective when only a limited number of labeled samples is available, e.g., l…
Data AugmentationSemantic SimilaritySemantic Textual Similaritytext-classification+1What Makes a Good Dataset for Knowledge Distillation?
Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher's original dataset will also be available when training the student. However, …
Continual LearningKnowledge DistillationModel CompressionWhat makes a good data augmentation for few-shot unsupervised image anomaly detection?
Data augmentation is a promising technique for unsupervised anomaly detection in industrial applications, where the availability of positive samples is often limited due to factors such as commercial competition and samp…
Anomaly DetectionData AugmentationUnsupervised Anomaly DetectionWhat makes instance discrimination good for transfer learning?
Contrastive visual pretraining based on the instance discrimination pretext task has made significant progress. Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for finetun…
object-detectionObject DetectionTransfer LearningTeaching What You Should Teach: A Data-Based Distillation Method
In real teaching scenarios, an excellent teacher always teaches what he (or she) is good at but the student is not. This gives the student the best assistance in making up for his (or her) weaknesses and becoming a good …
Data AugmentationKnowledge DistillationObject Recognition