Overparametrization improves robustness against adversarial attacks: A replication study
Overparametrization has become a de facto standard in machine learning. Despite numerous efforts, our understanding of how and where overparametrization helps model accuracy and robustness is still limited. To this end, here we conduct an empirical investigation to systemically study and replicate previous findings in this area, in particular the study by Madry et al. Together with this study, our findings support the "universal law of robustness" recently proposed by Bubeck et al. We argue that while critical for robust perception, overparametrization may not be enough to achieve full robustness and smarter architectures e.g. the ones implemented by the human visual cortex) seem inevitable.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning…
Adversarial AttackContrastive LearningLanguage ModellingSelf-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models
In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning…
Adversarial AttackContrastive LearningDual Manifold Adversarial Robustness: Defense against Lp and non-Lp Adversarial Attacks
Adversarial training is a popular defense strategy against attack threat models with bounded Lp norms. However, it often degrades the model performance on normal images and the defense does not generalize well to novel a…
Adversarial RobustnessThe curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression
Successful deep learning models often involve training neural network architectures that contain more parameters than the number of training samples. Such overparametrized models have been extensively studied in recent y…
regressionTowards Out-of-Distribution Adversarial Robustness
Adversarial robustness continues to be a major challenge for deep learning. A core issue is that robustness to one type of attack often fails to transfer to other attacks. While prior work establishes a theoretical trade…
Adversarial Robustness