Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
Instruction tuning enhances large vision-language models (LVLMs) but increases their vulnerability to backdoor attacks due to their open design. Unlike prior studies in static settings, this paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains. We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness under visual and text domain shifts. Our findings reveal two insights: (1) backdoor generalizability improves when distinctive trigger patterns are independent of specific data domains or model architectures, and (2) the competitive interaction between trigger patterns and clean semantic regions, where guiding the model to predict triggers enhances attack generalizability. Based on these insights, we propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas using attributional interpretation. Experiments with OpenFlamingo, Blip-2, and Otter show that MABA significantly boosts the attack success rate of generalization by 36.4%, achieving a 97% success rate at a 0.2% poisoning rate. This study reveals limitations in current evaluations and highlights how enhanced backdoor generalizability poses a security threat to LVLMs, even without test data access.
Code (0)
등록된 구현이 없습니다.
Tasks
Backdoor AttackDomain GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revisiting Personalized Federated Learning: Robustness Against Backdoor Attacks
In this work, besides improving prediction accuracy, we study whether personalization could bring robustness benefits to backdoor attacks. We conduct the first study of backdoor attacks in the pFL framework, testing 4 wi…
Backdoor AttackFederated LearningPersonalized Federated LearningOn Certifying Robustness against Backdoor Attacks via Randomized Smoothing
Backdoor attack is a severe security threat to deep neural networks (DNNs). We envision that, like adversarial examples, there will be a cat-and-mouse game for backdoor attacks, i.e., new empirical defenses are developed…
Backdoor AttackRevisiting the Assumption of Latent Separability for Backdoor Defenses
Recent studies revealed that deep learning is susceptible to backdoor poisoning attacks. An adversary can embed a hidden backdoor into a model to manipulate its predictions by only modifying a few training data, without …
Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks
Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive…
Backdoor AttackData PoisoningRepresentation Learningzero-shot-classification+1Robust Contrastive Language-Image Pretraining against Data Poisoning and Backdoor Attacks
Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive…