On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.
Code (1)
Tasks
SchedulingSimilar Papers 제목 키워드 기반
Adversarial Weight Perturbation Helps Robust Generalization
The study on improving the robustness of deep neural networks against adversarial examples grows rapidly in recent years. Among them, adversarial training is the most promising one, which flattens the input loss landscap…
Adversarial RobustnessAdversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression
Adversarial training is actively studied for learning robust models against adversarial examples. A recent study finds that adversarially trained models degenerate generalization performance on adversarial examples when …
regressionUnderstanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization
Adversarial training is widely used to improve the robustness of deep neural networks to adversarial attack. However, adversarial training is prone to overfitting, and the cause is far from clear. This work sheds light o…
Adversarial AttackRevisiting Video Quality Assessment from the Perspective of Generalization
The increasing popularity of short video platforms such as YouTube Shorts, TikTok, and Kwai has led to a surge in User-Generated Content (UGC), which presents significant challenges for the generalization performance of …
Image Quality AssessmentVideo Quality AssessmentVisual Question Answering (VQA)Learning to be adversarially robust and differentially private
We study the difficulties in learning that arise from robust and differentially private optimization. We first study convergence of gradient descent based adversarial training with differential privacy, taking a simple b…
Binary Classification