Semi-Online Knowledge Distillation
Knowledge distillation is an effective and stable method for model compression via knowledge transfer. Conventional knowledge distillation (KD) is to transfer knowledge from a large and well pre-trained teacher network to a small student network, which is a one-way process. Recently, deep mutual learning (DML) has been proposed to help student networks learn collaboratively and simultaneously. However, to the best of our knowledge, KD and DML have never been jointly explored in a unified framework to solve the knowledge distillation problem. In this paper, we investigate that the teacher model supports more trustworthy supervision signals in KD, while the student captures more similar behaviors from the teacher in DML. Based on these observations, we first propose to combine KD with DML in a unified framework. Furthermore, we propose a Semi-Online Knowledge Distillation (SOKD) method that effectively improves the performance of the student and the teacher. In this method, we introduce the peer-teaching training fashion in DML in order to alleviate the student's imitation difficulty, and also leverage the supervision signals provided by the well-trained teacher in KD. Besides, we also show our framework can be easily extended to feature-based distillation methods. Extensive experiments on CIFAR-100 and ImageNet datasets demonstrate the proposed method achieves state-of-the-art performance.
Code (1)
Tasks
Knowledge DistillationModel CompressionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning
The advancement of knowledge distillation has played a crucial role in enabling the transfer of knowledge from larger teacher models to smaller and more efficient student models, and is particularly beneficial for online…
Knowledge DistillationRoad SegmentationSemi-Supervised Learning with Online Knowledge Distillation for Skin Lesion Classification
Deep Learning has emerged as a promising approach for skin lesion analysis. However, existing methods mostly rely on fully supervised learning, requiring extensive labeled data, which is challenging and costly to obtain.…
Skin Lesion ClassificationKnowledge DistillationEnsemble LearningEfficient Personalized Reranking with Semi-Autoregressive Generation and Online Knowledge Distillation
Generative models offer a promising paradigm for the final stage reranking in multi-stage recommender systems, with the ability to capture inter-item dependencies within reranked lists. However, their practical deploymen…
Knowledge DistillationKnowledge Distillation as Semiparametric Inference
A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation proc…
Knowledge DistillationModel CompressionKnowledge Distillation with BERT for Image Tag-Based Privacy Prediction
Text in the form of tags associated with online images is often informative for predicting private or sensitive content from images. When using privacy prediction systems running on social networking sites that decide wh…
Knowledge DistillationTAG