RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Backdoor AttackData PoisoningRed TeamingSafety AlignmentMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning
Due to the broad range of applications of multi-agent reinforcement learning (MARL), understanding the effects of adversarial attacks against MARL model is essential for the safe applications of this model. Motivated by …
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningAdaptive Reward-Poisoning Attacks against Reinforcement Learning
In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward $r_t$ into $r_t+\delta_t$ at each step, with the goal of forcing the RL agent to learn a nefarious policy. W…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
We study black-box reward poisoning attacks against reinforcement learning (RL), in which an adversary aims to manipulate the rewards to mislead a sequence of RL agents with unknown algorithms to learn a nefarious policy…
reinforcement-learningReinforcement Learning (RL)Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learning
We study reward poisoning attacks on online deep reinforcement learning (DRL), where the attacker is oblivious to the learning algorithm used by the agent and the dynamics of the environment. We demonstrate the intrinsic…
Data PoisoningDeep Reinforcement LearningMuJoCoreinforcement-learning+2Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning
To understand the security threats to reinforcement learning (RL) algorithms, this paper studies poisoning attacks to manipulate \emph{any} order-optimal learning algorithm towards a targeted policy in episodic RL and ex…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)