How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?
With the rise of powerful foundation models, a pre-training-fine-tuning paradigm becomes increasingly popular these days: A foundation model is pre-trained using a huge amount of data from various sources, and then the downstream users only need to fine-tune and adapt it to specific downstream tasks. However, due to the high computation complexity of adversarial training, it is not feasible to fine-tune the foundation model to improve its robustness on the downstream task. Observing the above challenge, we want to improve the downstream robustness without updating/accessing the weights in the foundation model. Inspired from existing literature in robustness inheritance (Kim et al., 2020), through theoretical investigation, we identify a close relationship between robust contrastive learning with the adversarial robustness of supervised learning. To further validate and utilize this theoretical insight, we design a simple-yet-effective robust auto-encoder as a data pre-processing method before feeding the data into the foundation model. The proposed approach has zero access to the foundation model when training the robust auto-encoder. Extensive experiments demonstrate the effectiveness of the proposed method in improving the robustness of downstream tasks, verifying the connection between the feature robustness (implied by small adversarial contrastive loss) and the robustness of the downstream task.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessContrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders
The widespread use of publicly available pre-trained encoders from self-supervised learning (SSL) has exposed a critical vulnerability: their susceptibility to downstream-agnostic adversarial examples (DAEs), which are c…
Self-Supervised LearningAdversarial RobustnessAdversarial DefenseWhen Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?
Contrastive learning (CL) can learn generalizable feature representations and achieve the state-of-the-art performance of downstream tasks by finetuning a linear classifier on top of it. However, as adversarial robustnes…
Adversarial RobustnessContrastive Learningimage-classificationImage Classification+1Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness
The security and robustness of deep neural networks (DNNs) have become increasingly concerning. This paper aims to provide both a theoretical foundation and a practical solution to ensure the reliability of DNNs. We expl…
Adversarial RobustnessA Prompting-based Approach for Adversarial Example Generation and Robustness Enhancement
Recent years have seen the wide application of NLP models in crucial areas such as finance, medical treatment, and news media, raising concerns of the model robustness and vulnerabilities. In this paper, we propose a nov…
Adversarial AttackBetter Representations via Adversarial Training in Pre-Training: A Theoretical Perspective
Pre-training is known to generate universal representations for downstream tasks in large-scale deep learning such as large language models. Existing literature, e.g., \cite{kim2020adversarial}, empirically observe that …
Adversarial RobustnessContrastive Learningvalid