Understanding Zero-Shot Adversarial Robustness for Large-Scale Models

2022-12-14 · Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, Carl Vondrick

Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks. Yet imperceptible adversarial perturbations can significantly reduce CLIP's performance on new tasks. In this work, we identify and explore the problem of \emph{adapting large-scale models for zero-shot adversarial robustness}. We first identify two key factors during model adaption -- training losses and adaptation methods -- that affect the model's zero-shot adversarial robustness. We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data. We apply this training loss to two adaption methods, model finetuning and visual prompt tuning. We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance. Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets. We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models.

📄 PDF Abstract BibTeX arXiv:2212.07016

Code (2)

cvlab-columbia/zsrobust4foundationmodel 공식 구현 pytorch

chs20/robustvlm pytorch

Tasks

Adversarial Robustness Contrastive Learning Visual Prompt Tuning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Contrastive Learning 설명 없음

Language-Driven Anchors for Zero-Shot Adversarial Robustness

2023-01-30 · CVPR 2024 1 · Xiao Li, Wei zhang, Yining Liu, Zhanhao Hu 외

Deep Neural Networks (DNNs) are known to be susceptible to adversarial attacks. Previous researches mainly focus on improving adversarial robustness in the fully supervised setting, leaving the challenging domain of zero…

Adversarial Defense Adversarial Robustness Image Classification Zero-Shot Image Classification+1

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

2024-05-27 · Fengji Ma, Li Liu, Hei Victor Cheng

This work addresses the challenge of achieving zero-shot adversarial robustness while preserving zero-shot generalization in large-scale foundation models, with a focus on the popular Contrastive Language-Image Pre-train…

Adversarial Robustness Knowledge Distillation Zero-shot Generalization

A Deep Dive into Adversarial Robustness in Zero-Shot Learning

2020-08-17 · Mehmet Kerim Yucel, Ramazan Gokberk Cinbis, Pinar Duygulu

Machine learning (ML) systems have introduced significant advances in various fields, due to the introduction of highly complex models. Despite their success, it has been shown multiple times that machine learning models…

Adversarial Robustness BIG-bench Machine Learning Generalized Zero-Shot Learning Zero-Shot Learning

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

2026-03-31 · Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang arxiv

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-train…

Zero-shot Generalization Adversarial Robustness

Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness

2024-01-09 · CVPR 2024 1 · Sibo Wang, Jie Zhang, Zheng Yuan, Shiguang Shan

Large-scale pre-trained vision-language models like CLIP have demonstrated impressive performance across various tasks, and exhibit remarkable zero-shot generalization capability, while they are also vulnerable to imperc…

Adversarial Robustness Zero-shot Generalization