SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
Adversarial training (AT) has become a popular choice for training robust networks. However, it tends to sacrifice clean accuracy heavily in favor of robustness and suffers from a large generalization error. To address these concerns, we propose Smooth Adversarial Training (SAT), guided by our analysis on the eigenspectrum of the loss Hessian. We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric. We present a general formulation for curriculum learning in the adversarial setting and propose two difficulty metrics based on the maximal Hessian eigenvalue (H-SAT) and the softmax probability (P-SA). We demonstrate that SAT stabilizes network training even for a large perturbation norm and allows the network to operate at a better clean accuracy versus robustness trade-off curve compared to AT. This leads to a significant improvement in both clean accuracy and robustness compared to AT, TRADES, and other baselines. To highlight a few results, our best model improves normal and robust accuracy by 6% and 1% on CIFAR-100 compared to AT, respectively. On Imagenette, a ten-class subset of ImageNet, our model outperforms AT by 23% and 3% on normal and robust accuracy respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding
Curriculum learning and self-paced learning are the training strategies that gradually feed the samples from easy to more complex. They have captivated increasing attention due to their excellent performance in robotic v…
Multi-Label ClassificationMUlTI-LABEL-ClASSIFICATIONScene UnderstandingSemantic SegmentationFlatness-aware Curriculum Learning via Adversarial Difficulty
Neural networks trained by empirical risk minimization often suffer from overfitting, especially to specific samples or domains, which leads to poor generalization. Curriculum Learning (CL) addresses this issue by select…
Domain GeneralizationImage ClassificationAdversarial Robustness via Label-Smoothing
We study Label-Smoothing as a means for improving adversarial robustness of supervised deep-learning models. After establishing a thorough and unified framework, we propose several variations to this general method: adve…
Adversarial RobustnessAdversarial Robustness via Adaptive Label Smoothing
Adversarial training (AT) has become a dominant defense paradigm by enforcing the model's predictions to be locally invariant to adversarial examples. Being a simple technique, Label smoothing (LS) has shown its potentia…
Adversarial RobustnessUnsupervised Medical Image Alignment with Curriculum Learning
We explore different curriculum learning methods for training convolutional neural networks on the task of deformable pairwise 3D medical image registration. To the best of our knowledge, we are the first to attempt to i…
Image RegistrationMedical Image Registration