Conditional Generative Data-free Knowledge Distillation
Knowledge distillation has made remarkable achievements in model compression. However, most existing methods require the original training data, which is usually unavailable due to privacy and security issues. In this paper, we propose a conditional generative data-free knowledge distillation (CGDD) framework for training lightweight networks without any training data. This method realizes efficient knowledge distillation based on conditional image generation. Specifically, we treat the preset labels as ground truth to train a conditional generator in a semi-supervised manner. The trained generator can produce specified classes of training images. For training the student network, we force it to extract the knowledge hidden in teacher feature maps, which provide crucial cues for the learning process. Moreover, an adversarial training framework for promoting distillation performance is constructed by designing several loss functions. This framework helps the student model to explore larger data space. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on different datasets. Compared with other data-free works, our work obtains state-of-the-art results on CIFAR100, Caltech101, and different versions of ImageNet datasets. The codes will be released.
Code (0)
등록된 구현이 없습니다.
Tasks
Conditional Image GenerationData-free Knowledge DistillationImage GenerationKnowledge DistillationModel CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
Data-free knowledge distillation~(DFKD) is an effective manner to solve model compression and transmission restrictions while retaining privacy protection, which has attracted extensive attention in recent years. Current…
Data-free Knowledge DistillationContrastive LearningModel CompressionA Unified Knowledge Distillation Framework for Deep Directed Graphical Models
Knowledge distillation (KD) is a technique that transfers the knowledge from a large teacher network to a small student network. It has been widely applied to many different tasks, such as model compression and federate…
Continual LearningFederated LearningKnowledge DistillationModel CompressionLarge-Scale Generative Data-Free Distillation
Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning. Most existing distillation approaches require the access to original or a…
Knowledge DistillationModel CompressionTransfer LearningFedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
While existing federated learning approaches primarily focus on aggregating local models to construct a global model, in realistic settings, some clients may be reluctant to share their private models due to the inclusio…
Data-free Knowledge DistillationFederated LearningGenerative Adversarial NetworkKnowledge DistillationRandom Conditioning for Diffusion Model Compression with Distillation
Diffusion models generate high-quality images through progressive denoising but are computationally intensive due to large model sizes and repeated sampling. Knowledge distillation--transferring knowledge from a comp…
DenoisingKnowledge DistillationmodelModel Compression