BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP
Contrastive Vision-Language Pre-training, known as CLIP, has shown promising effectiveness in addressing downstream image recognition tasks. However, recent works revealed that the CLIP model can be implanted with a downstream-oriented backdoor. On downstream tasks, one victim model performs well on clean samples but predicts a specific target class whenever a specific trigger is present. For injecting a backdoor, existing attacks depend on a large amount of additional data to maliciously fine-tune the entire pre-trained CLIP model, which makes them inapplicable to data-limited scenarios. In this work, motivated by the recent success of learnable prompts, we address this problem by injecting a backdoor into the CLIP model in the prompt learning stage. Our method named BadCLIP is built on a novel and effective mechanism in backdoor attacks on CLIP, i.e., influencing both the image and text encoders with the trigger. It consists of a learnable trigger applied to images and a trigger-aware context generator, such that the trigger can change text features via trigger-aware prompts, resulting in a powerful and generalizable attack. Extensive experiments conducted on 11 datasets verify that the clean accuracy of BadCLIP is similar to those of advanced prompt learning methods and the attack success rate is higher than 99% in most cases. BadCLIP is also generalizable to unseen classes, and shows a strong generalization capability under cross-dataset and cross-domain settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Prompt LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning
Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence. Existing methods often fail under strong detection or continuous fine-tuning, largely du…
Contrastive LearningBadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be eas…
Backdoor AttackContrastive LearningBackdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning
Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts, enabling strong zero-shot generalization to novel concepts. As these models gain traction in…
Zero-shot GeneralizationAutonomous DrivingCross-Paradigm Graph Backdoor Attacks with Promptable Subgraph Triggers
Graph Neural Networks(GNNs) are vulnerable to backdoor attacks, where adversaries implant malicious triggers to manipulate model predictions. Existing trigger generators are often simplistic in structure and overly relia…
Contrastive LearningGraph LearningFrom Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
Large Language Models (LLMs) can acquire deceptive behaviors through backdoor attacks, where the model executes prohibited actions whenever secret triggers appear in the input. Existing safety training methods largely fa…
Reinforcement Learning