BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF
Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning text-to-image (T2I) models with human preferences. However, RLHF's feedback mechanism also opens new pathways for adversaries. This paper demonstrates the feasibility of hijacking T2I models by poisoning a small fraction of preference data with natural-appearing examples. Specifically, we propose BadReward, a stealthy clean-label poisoning attack targeting the reward model in multi-modal RLHF. BadReward operates by inducing feature collisions between visually contradicted preference data instances, thereby corrupting the reward model and indirectly compromising the T2I model's integrity. Unlike existing alignment poisoning techniques focused on single (text) modality, BadReward is independent of the preference annotation process, enhancing its stealth and practical threat. Extensive experiments on popular T2I models show that BadReward can consistently guide the generation towards improper outputs, such as biased or violent imagery, for targeted concepts. Our findings underscore the amplified threat landscape for RLHF in multi-modal systems, highlighting the urgent need for robust defenses. Disclaimer. This paper contains uncensored toxic content that might be offensive or disturbing to the readers.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Certified Robustness to Clean-Label Poisoning Using Diffusion Denoising
We present a certified defense to clean-label poisoning attacks under $\ell_2$-norm. These attacks work by injecting a small number of poisoning samples (e.g., 1%) that contain bounded adversarial perturbations into the …
Adversarial RobustnessDenoisingDeep k-NN Defense against Clean-label Data Poisoning Attacks
Targeted clean-label data poisoning is a type of adversarial attack on machine learning systems in which an adversary injects a few correctly-labeled, minimally-perturbed samples into the training data, causing a model t…
Adversarial AttackData PoisoningAdversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systems
Clean-label (CL) attack is a form of data poisoning attack where an adversary modifies only the textual input of the training data, without requiring access to the labeling function. CL attacks are relatively unexplored …
Data Poisoningtext-classificationText ClassificationMetaPoison: Practical General-purpose Clean-label Data Poisoning
Data poisoning -- the process by which an attacker takes control of a model by making imperceptible changes to a subset of the training data -- is an emerging threat in the context of neural networks. Existing attacks fo…
AutoMLBilevel OptimizationData PoisoningMeta-LearningCheckerboard: Closed-Form and Data-Independent Trigger Design for Clean-Label Backdoor Attacks
Backdoor attacks threaten the deep-learning supply chain by poisoning a small fraction of the training data so that a model behaves normally on clean inputs but maps triggered inputs to an attacker-chosen class. Clean-la…