Improved OOD Generalization via Adversarial Training and Pre-training
Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we theoretically show that adversarially trained models have converged excess risk on OOD data, and empirically verify it on both image classification and natural language understanding tasks. Besides, in the paradigm of first pre-training and then fine-tuning, we theoretically show that a pre-trained model that is more robust to input perturbation provides a better initialization for generalization on downstream OOD data. Empirically, after fine-tuning, this better-initialized model from adversarial pre-training also has better OOD generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationNatural Language UnderstandingSimilar Papers 제목 키워드 기반
Stability and Generalization in Free Adversarial Training
While adversarial training methods have significantly improved the robustness of deep neural networks against norm-bounded adversarial perturbations, the generalization gap between their performance on training and test …
Generalization BoundsImpact of Adversarial Training on Robustness and Generalizability of Language Models
Adversarial training is widely acknowledged as the most effective defense against adversarial attacks. However, it is also well established that achieving both robustness and generalization in adversarially trained model…
Data AugmentationCAT: Customized Adversarial Training for Improved Robustness
Adversarial training has become one of the most effective methods for improving robustness of neural networks. However, it often suffers from poor generalization on both clean and perturbed data. In this paper, we propos…
Improving the Generalization of Adversarial Training with Domain Adaptation
By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models. However, most existing adversarial training approaches are based on a specific…
Adversarial AttackDomain AdaptationGenerating Less Certain Adversarial Examples Improves Robust Generalization
This paper revisits the robust overfitting phenomenon of adversarial training. Observing that models with better robust generalization performance are less certain in predicting adversarially generated training inputs, w…