Data-Free Knowledge Distillation for Deep Neural Networks
Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.
Code (2)
Tasks
Data-free Knowledge DistillationKnowledge DistillationModel CompressionSimilar Papers 제목 키워드 기반
Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
Distilling knowledge from huge pre-trained networks to improve the performance of tiny networks has favored deep learning models to be used in many real-time and mobile applications. Several approaches that demonstrate s…
Data-free Knowledge DistillationDiversityKnowledge DistillationSegmentationSmall Scale Data-Free Knowledge Distillation
Data-free knowledge distillation is able to utilize the knowledge learned by a large teacher network to augment the training of a smaller student network without accessing the original training data, avoiding privacy, se…
Data-free Knowledge DistillationGenerative Adversarial Networkimage-classificationImage Classification+2Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
Knowledge distillation refers to a technique of transferring the knowledge from a large learned model or an ensemble of learned models to a small model. This method relies on access to the original training set, which mi…
Knowledge DistillationAdversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks
Data-free Knowledge Distillation (DFKD) is a method that constructs pseudo-samples using a generator without real data, and transfers knowledge from a teacher model to a student by enforcing the student to overcome dimen…
Data-free Knowledge DistillationKnowledge DistillationAdversarial Self-Supervised Data-Free Distillation for Text Classification
Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computation…
ClassificationGeneral ClassificationKnowledge Distillationtext-classification+2