Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

2024-01-30 · Andy Zhou, Bo Li, Haohan Wang

Despite advances in AI alignment, large language models (LLMs) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries can modify prompts to induce unwanted behavior. While some defenses have been proposed, they have not been adapted to newly proposed attacks and more challenging threat models. To address this, we propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm, Robust Prompt Optimization (RPO) to create robust system-level defenses. Our approach directly incorporates the adversary into the defensive objective and optimizes a lightweight and transferable suffix, enabling RPO to adapt to worst-case adaptive attacks. Our theoretical and experimental results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench, setting the state-of-the-art. Code can be found at https://github.com/lapisrocks/rpo

📄 PDF Abstract BibTeX arXiv:2401.17263

Code (1)

lapisrocks/rpo 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Multi-Head Attention 설명 없음

Adam 설명 없음

Defending LLMs against Jailbreaking Attacks via Backtranslation

2024-02-26 · Yihan Wang, Zhouxing Shi, Andrew Bai, Cho-Jui Hsieh

Although many large language models (LLMs) have been trained to refuse harmful requests, they are still vulnerable to jailbreaking attacks which rewrite the original prompt to conceal its harmful intent. In this paper, w…

Language Modelling

Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM

2023-09-18 · Bochuan Cao, Yuanpu Cao, Lu Lin, Jinghui Chen

Recently, Large Language Models (LLMs) have made significant advancements and are now widely used across various domains. Unfortunately, there has been a rising concern that LLMs can be misused to generate harmful or mal…

SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

2023-10-05 · Alexander Robey, Eric Wong, Hamed Hassani, George J. Pappas

Despite efforts to align large language models (LLMs) with human intentions, widely-used LLMs such as GPT, Llama, and Claude are susceptible to jailbreaking attacks, wherein an adversary fools a targeted LLM into generat…

Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts

2023-11-15 · Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou 외

Existing work on jailbreak Multimodal Large Language Models (MLLMs) has focused primarily on adversarial examples in model inputs, with less attention to vulnerabilities, especially in model API. To fill the research gap…

Adversarial Attack Red Teaming

Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval

2025-05-21 · Taiye Chen, Zeming Wei, Ang Li, Yisen Wang

Large Language Models (LLMs) are known to be vulnerable to jailbreaking attacks, wherein adversaries exploit carefully engineered prompts to induce harmful or unethical responses. Such threats have raised critical concer…

RAG Retrieval Retrieval-augmented Generation