Customizing Synthetic Data for Data-Free Student Learning

2023-07-10 · Shiya Luo, Defang Chen, Can Wang

Data-free knowledge distillation (DFKD) aims to obtain a lightweight student model without original training data. Existing works generally synthesize data from the pre-trained teacher model to replace the original training data for student learning. To more effectively train the student model, the synthetic data shall be customized to the current student learning ability. However, this is ignored in the existing DFKD methods and thus negatively affects the student training. To address this issue, we propose Customizing Synthetic Data for Data-Free Student Learning (CSD) in this paper, which achieves adaptive data synthesis using a self-supervised augmented auxiliary task to estimate the student learning ability. Specifically, data synthesis is dynamically adjusted to enlarge the cross entropy between the labels and the predictions from the self-supervised augmented task, thus generating hard samples for the student model. The experiments on various datasets and teacher-student models show the effectiveness of our proposed method. Code is available at: $\href{https://github.com/luoshiya/CSD}{https://github.com/luoshiya/CSD}$

📄 PDF Abstract BibTeX arXiv:2307.04542

Code (1)

luoshiya/csd 공식 구현 pytorch

Tasks

Data-free Knowledge Distillation Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Fair4Free: Generating High-fidelity Fair Synthetic Samples using Data Free Distillation

2024-10-02 · Md Fahim Sikder, Daniel de Leng, Fredrik Heintz

This work presents Fair4Free, a novel generative model to generate synthetic fair data using data-free distillation in the latent space. Fair4Free can work on the situation when the data is private or inaccessible. In ou…

Fairness

Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

2024-10-24 · Anup Shirgaonkar, Nikhil Pandey, Nazmiye Ceren Abay, Tolga Aktas 외

Leading open-source large language models (LLMs) such as Llama-3.1-Instruct-405B are extremely capable at generating text, answering questions, and solving a variety of natural language understanding tasks. However, they…

Knowledge Distillation Natural Language Understanding

Synthetic data generation method for data-free knowledge distillation in regression neural networks

2023-01-11 · Tianxun Zhou, Keng-Hwee Chiam

Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neur…

Data-free Knowledge Distillation Knowledge Distillation regression Synthetic Data Generation

Privacy-Preserving Student Learning with Differentially Private Data-Free Distillation

2024-09-19 · Bochao Liu, Jianghu Lu, Pengju Wang, Junjie Zhang 외

Deep learning models can achieve high inference accuracy by extracting rich knowledge from massive well-annotated data, but may pose the risk of data privacy leakage in practical deployment. In this paper, we present an …

Privacy Preserving Privacy Preserving Deep Learning

Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay

2022-01-09 · Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman 외

Data-Free Knowledge Distillation (KD) allows knowledge transfer from a trained neural network (teacher) to a more compact one (student) in the absence of original training data. Existing works use a validation set to mon…

Data-free Knowledge Distillation image-classification Image Classification Knowledge Distillation+1