You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations, which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to the unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin's Maximal Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (You Only Propagate Once). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with approximately 1/5 ~ 1/4 GPU time of the projected gradient descent (PGD) algorithm. Our codes are available at https://https://github.com/a1600012888/YOPO-You-Only-Propagate-Once.
Code (2)
Tasks
Adversarial DefenseGPUSimilar Papers 제목 키워드 기반
Semi-Implicit Hybrid Gradient Methods with Application to Adversarial Robustness
Adversarial examples, crafted by adding imperceptible perturbations to natural inputs, can easily fool deep neural networks (DNNs). One of the most successful methods for training adversarially robust DNNs is solving a n…
Adversarial RobustnessErosion Attack for Adversarial Training to Enhance Semantic Segmentation Robustness
Existing segmentation models exhibit significant vulnerability to adversarial attacks.To improve robustness, adversarial training incorporates adversarial examples into model training. However, existing attack methods co…
Semantic SegmentationAdversarial training for tabular data with attack propagation
Adversarial attacks are a major concern in security-centered applications, where malicious actors continuously try to mislead Machine Learning (ML) models into wrongly classifying fraudulent activity as legitimate, where…
Feature EngineeringFraud DetectionAccelerating Optimization via Differentiable Stopping Time
Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This e…
Addressing The False Negative Problem of MRI Reconstruction Networks by Adversarial Attacks and Robust Training
Deep learning models have been shown to be successful in accelerating MRI reconstruction, over traditional methods. However, it has been observed that these methods tend to miss rare small features, such as meniscal tear…
DiagnosticMRI ReconstructionSSIM