Adversarially Robust Distillation
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.
Code (2)
Tasks
Adversarial RobustnessKnowledge DistillationSimilar Papers 제목 키워드 기반
Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
Adversarial training is one effective approach for training robust deep neural networks against adversarial attacks. While being able to bring reliable robustness, adversarial training (AT) methods in general favor high …
Adversarial RobustnessKnowledge DistillationDynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge
In the realm of Adversarial Distillation (AD), strategic and precise knowledge transfer from an adversarially robust teacher model to a less robust student model is paramount. Our Dynamic Guidance Adversarial Distillatio…
Transfer LearningImproving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks, crafted perturbations designed to mislead predictions. Despite advances in adversarial training, a gap persists be…
Adversarial RobustnessKnowledge DistillationReleasing Inequality Phenomena in $L_{\infty}$-Adversarial Training via Input Gradient Distillation
Since adversarial examples appeared and showed the catastrophic degradation they brought to DNN, many adversarial defense methods have been devised, among which adversarial training is considered the most effective. Howe…
Adversarial DefenseAdversarial RobustnessKDGAN: Knowledge Distillation with Generative Adversarial Networks
Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classif…
Knowledge DistillationMulti-Label Learning