Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?

2023-05-22 · Zheng Li, YuXuan Li, Penghai Zhao, RenJie Song, Xiang Li, Jian Yang

Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images. Given their huge success, it is still unclear whether synthetic images are applicable for knowledge distillation when real images are unavailable. In this paper, we extensively study whether and how synthetic images produced from state-of-the-art diffusion models can be used for knowledge distillation without access to real images, and obtain three key conclusions: (1) synthetic data from diffusion models can easily lead to state-of-the-art performance among existing synthesis-based distillation methods, (2) low-fidelity synthetic images are better teaching materials, and (3) relatively weak classifiers are better teachers. Code is available at https://github.com/zhengli97/DM-KD.

📄 PDF Abstract BibTeX arXiv:2305.12954

Code (1)

zhengli97/dm-kd 공식 구현 pytorch

Tasks

Data-free Knowledge Distillation Few-Shot Learning Knowledge Distillation Mitigating Contextual Bias

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

DiffKD-DCIS: Predicting Upgrade of Ductal Carcinoma In Situ with Diffusion Augmentation and Knowledge Distillation

2026-01-04 · Tao Li, Qing Li, Na Li, Hui Xie arxiv

Accurately predicting the upgrade of ductal carcinoma in situ (DCIS) to invasive ductal carcinoma (IDC) is crucial for surgical planning. However, traditional deep learning methods face challenges due to limited ultrasou…

Computational Efficiency Knowledge Distillation Data Augmentation

Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models

2023-06-02 · Virginia Fernandez, Pedro Sanchez, Walter Hugo Lopez Pinaya, Grzegorz Jacenków 외

Knowledge distillation in neural networks refers to compressing a large model or dataset into a smaller version of itself. We introduce Privacy Distillation, a framework that allows a text-to-image generative model to te…

Knowledge Distillation

DeepBridge: A Unified and Production-Ready Framework for Multi-Dimensional Machine Learning Validation

2025-12-18 · Gustavo Coelho Haase, Paulo Henrique Dourado da Silva arxiv

We present DeepBridge, an 80K-line Python library that unifies multi-dimensional validation, automatic compliance verification, knowledge distillation, and synthetic data generation. DeepBridge offers: (i) 5 validation s…

Synthetic Data Generation Knowledge Distillation

SER-Diff: Synthetic Error Replay Diffusion for Incremental Brain Tumor Segmentation

2025-10-06 · Sashank Makanaboyina arxiv

Incremental brain tumor segmentation is critical for models that must adapt to evolving clinical datasets without retraining on all prior data. However, catastrophic forgetting, where models lose previously acquired know…

Brain Tumor Segmentation Knowledge Distillation Incremental Learning

DerMAE: Improving skin lesion classification through conditioned latent diffusion and MAE distillation

2026-02-23 · Francisco Filho, Kelvin Cunha, Fábio Papais, Emanoel dos Santos 외 arxiv

Skin lesion classification datasets often suffer from severe class imbalance, with malignant cases significantly underrepresented, leading to biased decision boundaries during deep learning training. We address this chal…

Skin Lesion Classification Knowledge Distillation