Large-Scale Generative Data-Free Distillation

2020-12-10 · Liangchen Luo, Mark Sandler, Zi Lin, Andrey Zhmoginov, Andrew Howard

Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning. Most existing distillation approaches require the access to original or augmented training samples. But this can be problematic in practice due to privacy, proprietary and availability concerns. Recent work has put forward some methods to tackle this problem, but they are either highly time-consuming or unable to scale to large datasets. To this end, we propose a new method to train a generative image model by leveraging the intrinsic normalization layers' statistics of the trained teacher network. This enables us to build an ensemble of generators without training data that can efficiently produce substitute inputs for subsequent distillation. The proposed method pushes forward the data-free distillation performance on CIFAR-10 and CIFAR-100 to 95.02% and 77.02% respectively. Furthermore, we are able to scale it to ImageNet dataset, which to the best of our knowledge, has never been done using generative models in a data-free setting.

📄 PDF Abstract BibTeX arXiv:2012.05578

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Model Compression Transfer Learning

Small Scale Data-Free Knowledge Distillation

2024-06-12 · CVPR 2024 1 · He Liu, Yikai Wang, Huaping Liu, Fuchun Sun 외

Data-free knowledge distillation is able to utilize the knowledge learned by a large teacher network to augment the training of a smaller student network without accessing the original training data, avoiding privacy, se…

Data-free Knowledge Distillation Generative Adversarial Network image-classification Image Classification+2

PRISM: Precision-Recall Informed Data-Free Knowledge Distillation via Generative Diffusion

2025-09-21 · Xuewan He, Jielei Wang, Zihan Cheng, Yuchen Su 외 arxiv

Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access to the real in-distribution (ID) data. While existing methods perform well on small-scale images, they suffer from mo…

Data-free Knowledge Distillation Domain Generalization Prompt Engineering

Adversarial Score Distillation: When score distillation meets GAN

2023-12-01 · CVPR 2024 1 · Min Wei, Jingkai Zhou, Junyao Sun, Xuesong Zhang

Existing score distillation methods are sensitive to classifier-free guidance (CFG) scale: manifested as over-smoothness or instability at small CFG scales, while over-saturation at large ones. To explain and analyze the…

Generative Adversarial Network Text to 3D

ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation

2026-02-26 · Ayush Roy, Wei-Yang Alex Lee, Rudrasis Chakraborty, Vishnu Suresh Lokhande arxiv

In recent times, large datasets hinder efficient model training while also containing redundant concepts. Dataset distillation aims to synthesize compact datasets that preserve the knowledge of large-scale training sets …

Data-Free Knowledge Distillation via Feature Exchange and Activation Region Constraint

2023-01-01 · CVPR 2023 1 · Shikang Yu, Jiachen Chen, Hu Han, Shuqiang Jiang

Despite the tremendous progress on data-free knowledge distillation (DFKD) based on synthetic data generation, there are still limitations in diverse and efficient data synthesis. It is naive to expect that a simple …

Data Augmentation Data-free Knowledge Distillation Knowledge Distillation Synthetic Data Generation