R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation
Test-time adaptation (TTA) has emerged as a promising solution to tackle the continual domain shift in machine learning by allowing model parameters to change at test time, via self-supervised learning on unlabeled testing data. At the same time, it unfortunately opens the door to unforeseen vulnerabilities for degradation over time. Through a simple theoretical continual TTA model, we successfully identify a risk in the sampling process of testing data that could easily degrade the performance of a continual TTA model. We name this risk as Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a result of the unintended query from general TTA users. The risk posed by RIP is also highly realistic, as it does not require prior knowledge of model parameters or modification of testing samples. This simple requirement makes RIP as the first black-box TTA attack algorithm that stands out from existing white-box attempts. We extensively benchmark the performance of the most recent continual TTA approaches when facing the RIP attack, providing insights on its success, and laying out potential roadmaps that could enhance the resilience of future continual TTA systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningTest-time AdaptationSimilar Papers 제목 키워드 기반
Black-Box Continual Learning for Vision-Language Models
The rapid deployment of Vision-Language Models (VLMs) in dynamic environments necessitates the ability to learn continuously without forgetting. However, traditional continual learning (CL) settings often rely on white-b…
Continual LearningSimple Post-Training Robustness Using Test Time Augmentations and Random Forest
Although Deep Neural Networks (DNNs) achieve excellent performance on many real-world tasks, they are highly vulnerable to adversarial attacks. A leading defense against such attacks is adversarial training, a technique …
Adversarial RobustnessDiversityAgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
Multimodal agents built on large vision-language models (LVLMs) are increasingly deployed in open-world settings but remain highly vulnerable to prompt injection, especially through visual inputs. We introduce AgentTypo,…
Continual LearningData Poisoning Attack Aiming the Vulnerability of Continual Learning
Generally, regularization-based continual learning models limit access to the previous task data to imitate the real-world constraints related to memory and privacy. However, this introduces a problem in these models by …
Adversarial AttackContinual LearningData PoisoningBlack-box Adversarial Attacks in Autonomous Vehicle Technology
Despite the high quality performance of the deep neural network in real-world applications, they are susceptible to minor perturbations of adversarial attacks. This is mostly undetectable to human vision. The impact of s…
Autonomous VehiclesTraffic Sign Recognition