Robustifying Vision-Language Models via Test-Time Prompt Adaptation

2026-07-10 · Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu, Jiannan Ge, Jiaheng Zhang, Long Chen arxiv

Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.

📄 PDF Abstract BibTeX arXiv:2607.09450

Code (0)

등록된 구현이 없습니다.

Tasks

Zero-shot Generalization Adversarial Robustness Test-time Adaptation

Robustifying Language Models with Test-Time Adaptation

2023-10-29 · Noah Thomas McDermott, Junfeng Yang, Chengzhi Mao

Large-scale language models achieved state-of-the-art performance over a number of language tasks. However, they fail on adversarial language examples, which are sentences optimized to fool the language models but with s…

Language Modeling Language Modelling Sentence Sentence Classification+1

In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model

2024-03-10 · Junhui Yin, Xinyu Zhang, Lin Wu, Xiaojie Wang

Current pre-trained vision-language models, such as CLIP, have demonstrated remarkable zero-shot generalization capabilities across various downstream tasks. However, their performance significantly degrades when test in…

In-Context Learning Language Modeling Language Modelling Prompt Learning+1

A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models

2024-05-23 · Mario Döbler, Robert A. Marsden, Tobias Raichle, Bin Yang

In deep learning, maintaining model robustness against distribution shifts is critical. This work explores a broad range of possibilities to adapt vision-language foundation models at test-time, with a particular emphasi…

Image Classification Prompt Engineering Prompt Learning Test-time Adaptation

MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models

2025-12-13 · Yuqing Lei, Yingjun Du, Yawen Huang, Xiantong Zhen 외 arxiv

Vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization but remain sensitive to domain shifts at test time. Test-time prompt tuning (TPT) mitigates this issue by adapting prompts with fixed aug…

Zero-shot Generalization Domain Generalization Test-time Adaptation

SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models

2023-09-21 · NeurIPS 2023 11

Test-time adaptation (TTA) is a special and practical setting in unsupervised domain adaptation, which allows a pre-trained model in a source domain to adapt to unlabeled test data in another target domain. To avoid the …