DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization

2024-08-18 · Pucheng Dang, Xing Hu, Dong Li, Rui Zhang, Qi Guo, Kaidi Xu

Current text-to-image (T2I) synthesis diffusion models raise misuse concerns, particularly in creating prohibited or not-safe-for-work (NSFW) images. To address this, various safety mechanisms and red teaming attack methods are proposed to enhance or expose the T2I model's capability to generate unsuitable content. However, many red teaming attack methods assume knowledge of the text encoders, limiting their practical usage. In this work, we rethink the case of \textit{purely black-box} attacks without prior knowledge of the T2l model. To overcome the unavailability of gradients and the inability to optimize attacks within a discrete prompt space, we propose DiffZOO which applies Zeroth Order Optimization to procure gradient approximations and harnesses both C-PRV and D-PRV to enhance attack prompts within the discrete prompt domain. We evaluated our method across multiple safety mechanisms of the T2I diffusion model and online servers. Experiments on multiple state-of-the-art safety mechanisms show that DiffZOO attains an 8.5% higher average attack success rate than previous works, hence its promise as a practical red teaming tool for T2l models.

📄 PDF Abstract BibTeX arXiv:2408.11071

Code (0)

등록된 구현이 없습니다.

Tasks

Red Teaming

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Query-Efficient Black-Box Red Teaming via Bayesian Optimization

2023-05-27 · Deokjae Lee, JunYeong Lee, Jung-Woo Ha, Jin-Hwa Kim 외

The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways. We focus on the problem of black-box red teaming, where a red team generates tes…

Bayesian Optimization Language Modeling Language Modelling Red Teaming

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

2024-08-27 · Yuhao Du, Zhuo Li, Pengyu Cheng, Xiang Wan 외

Despite the substantial advancements in artificial intelligence, large language models (LLMs) remain being challenged by generation safety. With adversarial jailbreaking prompts, one can effortlessly induce LLMs to outpu…

Prompt Engineering Red Teaming

Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search

2024-08-11 · Robert J. Moss

Eliciting harmful behavior from large language models (LLMs) is an important task to ensure the proper alignment and safety of the models. Often when training LLMs, ethical guidelines are followed yet alignment failures …

Red Teaming

Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Sampling

2026-05-18 · Ziwei Wang, Jing Chen, Ruichao Liang, Zhi Wang 외 arxiv

Despite rigorous safety alignment, Large Language Models (LLMs) remain vulnerable to jailbreak attacks. Existing black-box methods often rely on heuristic templates or exhaustive trials, lacking mechanistic interpretabil…

When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models

2025-10-09 · Haoran Ou, Kangjie Chen, Xingshuo Han, Gelei Deng 외 arxiv

Large Language Models (LLMs) have been augmented with web search to overcome the limitations of the static knowledge boundary by accessing up-to-date information from the open Internet. While this integration enhances mo…