MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition

2022-08-11 · Chuanguang Yang, Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, Qian Zhang

Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to learn knowledge from itself without any guidance from extra networks. This paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates these two techniques into a unified framework. MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way. Therefore, it guides the network to learn cross-image knowledge by modelling supervisory signals from mixup images. Moreover, we construct a self-teacher network by aggregating multi-stage feature maps for providing soft labels to supervise the backbone classifier, further improving the efficacy of self-boosting. Experiments on image classification and transfer learning to object detection and semantic segmentation demonstrate that MixSKD outperforms other state-of-the-art Self-KD and data augmentation methods. The code is available at https://github.com/winycg/Self-KD-Lib.

📄 PDF Abstract BibTeX arXiv:2208.05768

Code (1)

winycg/self-kd-lib 공식 구현 pytorch

Tasks

Data Augmentation image-classification Image Classification Knowledge Distillation object-detection Object DetectionSelf-Knowledge DistillationSemantic Segmentation Transfer Learning

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Mixup Mixup is a data augmentation technique that generates a weighted combination of random image pairs from the training data. Given two images and their ground truth labels:…

Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing

2025-06-26 · Lars Möllenbrok, Behnood Rasti, Begüm Demir

The development of continual learning (CL) methods, which aim to learn new tasks in a sequential manner from the training data acquired continuously, has gained great attention in remote sensing (RS). The existing CL met…

Continual Learning Continual Self-Supervised Learning Knowledge Distillation Self-Supervised Learning

Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study

2022-11-08 · Hongjun Choi, Eun Som Jeon, Ankita Shukla, Pavan Turaga

Mixup is a popular data augmentation technique based on creating new samples by linear interpolation between two given data samples, to improve both the generalization and robustness of the trained model. Knowledge disti…

Attribute Data Augmentation image-classification Image Classification+3

Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup

2020-12-17 · Guodong Xu, Ziwei Liu, Chen Change Loy

Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an essential technique for model compression and transfer learning.…

Informativeness Knowledge Distillation Model Compression Transfer Learning

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

2026-06-10 · José Medina, Paul Honeine, Abdelaziz Bensrhair, Amnir Hadachi arxiv

Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combin…

Knowledge Distillation

Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT

2022-07-01 · NAACL (GeBNLP) 2022 7 · Jaimeen Ahn, Hwaran Lee, JinHwa Kim, Alice Oh

Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model.However, after knowledge distillation, it was found that the smaller model is more biased by gender compare…