TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models

2023-06-12 · NeurIPS 2023 11 · Jiaqi Xue, Mengxin Zheng, Ting Hua, Yilin Shen, Yepeng Liu, Ladislau Boloni, Qian Lou

Large Language Models (LLMs) are progressively being utilized as machine learning services and interface tools for various applications. However, the security implications of LLMs, particularly in relation to adversarial and Trojan attacks, remain insufficiently examined. In this paper, we propose TrojLLM, an automatic and black-box framework to effectively generate universal and stealthy triggers. When these triggers are incorporated into the input data, the LLMs' outputs can be maliciously manipulated. Moreover, the framework also supports embedding Trojans within discrete prompts, enhancing the overall effectiveness and precision of the triggers' attacks. Specifically, we propose a trigger discovery algorithm for generating universal triggers for various inputs by querying victim LLM-based APIs using few-shot data samples. Furthermore, we introduce a novel progressive Trojan poisoning algorithm designed to generate poisoned prompts that retain efficacy and transferability across a diverse range of models. Our experiments and results demonstrate TrojLLM's capacity to effectively insert Trojans into text prompts in real-world black-box LLM APIs including GPT-3.5 and GPT-4, while maintaining exceptional performance on clean test sets. Our work sheds light on the potential security risks in current models and offers a potential defensive approach. The source code of TrojLLM is available at https://github.com/UCF-ML-Research/TrojLLM.

📄 PDF Abstract BibTeX arXiv:2306.06815

Code (1)

ucf-ml-research/trojllm 공식 구현 pytorch

TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning

2026-01-18 · Zhixin Xie, Xurui Song, Jun Luo arxiv

The demand of customized large language models (LLMs) has led to commercial LLMs offering black-box fine-tuning APIs, yet this convenience introduces a critical security loophole: attackers could jailbreak the LLMs by fi…

An Adaptive Black-box Defense against Trojan Attacks (TrojDef)

2022-09-05 · Guanxiong Liu, Abdallah Khreishah, Fatima Sharadgah, Issa Khalil

Trojan backdoor is a poisoning attack against Neural Network (NN) classifiers in which adversaries try to exploit the (highly desirable) model reuse property to implant Trojans into model parameters for backdoor breaches…

Prediction

Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settings

2024-12-18 · Yuanhe Zhang, Zhenhong Zhou, Wei zhang, Xinyue Wang 외

Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks yet still are vulnerable to external threats, particularly LLM Denial-of-Service (LLM-DoS) attacks. Specifically, LLM-DoS attacks…

GPU

Detecting AI Trojans Using Meta Neural Analysis

2019-10-08 · Xiaojun Xu, Qi. Wang, Huichen Li, Nikita Borisov 외

In machine learning Trojan attacks, an adversary trains a corrupted model that obtains good performance on normal data but behaves maliciously on data samples with certain trigger patterns. Several approaches have been p…

Data Poisoning

From Prompt Injection to Persistent Control: Defending Agentic Harness Against Trojan Backdoors

2026-05-29 · Jiejun Tan, Zhicheng Dou, Xinyu Yang, Yuyang Hu 외 arxiv

LLM agents are evolving from conversational chatbots to operational tools in real-world workspaces. In local agentic harnesses, an LLM can read and write files, call tools, and reuse workspace state across sessions. Whil…