Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images
Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the original training dataset is always available. However, this is not always the case due to privacy concerns and more. In recent years, "data-free" KD has emerged as a growing research topic which focuses on the scenario of performing KD when no data is provided. Many methods rely on a generator network to synthesize examples for distillation (which can be difficult to train) and can frequently produce images that are visually similar to the original dataset, which raises questions surrounding whether privacy is completely preserved. In this work, we propose a new approach to data-free KD that utilizes unnatural OpenGL images, combined with large amounts of data augmentation and adversarial attacks, to train a student network. We demonstrate that our approach achieves state-of-the-art results for a variety of datasets/networks and is more stable than existing generator-based data-free KD methods. Source code will be available in the future.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationData-free Knowledge DistillationKnowledge DistillationModel CompressionSimilar Papers 제목 키워드 기반
Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge
In the realm of Adversarial Distillation (AD), strategic and precise knowledge transfer from an adversarially robust teacher model to a less robust student model is paramount. Our Dynamic Guidance Adversarial Distillatio…
Transfer LearningImproving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks, crafted perturbations designed to mislead predictions. Despite advances in adversarial training, a gap persists be…
Adversarial RobustnessKnowledge DistillationDARD: Dice Adversarial Robustness Distillation against Adversarial Attacks
Deep learning models are vulnerable to adversarial examples, posing critical security challenges in real-world applications. While Adversarial Training (AT ) is a widely adopted defense mechanism to enhance robustness, i…
Adversarial RobustnessKnowledge DistillationSynthetic data generation method for data-free knowledge distillation in regression neural networks
Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neur…
Data-free Knowledge DistillationKnowledge DistillationregressionSynthetic Data GenerationAdversarially Robust Distillation
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness …
Adversarial RobustnessKnowledge Distillation