Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications
Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications such as code completion and AI-powered search. LLM-integrated applications serve as middleware to refine users' queries with domain-specific knowledge to better inform LLMs and enhance the responses. Despite numerous opportunities and benefits, LLM-integrated applications also introduce new attack surfaces. Understanding, minimizing, and eliminating these emerging attack surfaces is a new area of research. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We focus on the communication rounds that begin with user's queries and end with LLM-integrated application returning responses to the queries, powered by LLMs at the service backend. For this query-response protocol, we identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat initiator that is able to control the database access, manipulate and poison data that are high-risk for the user. Successful exploits of the identified vulnerabilities result in the users receiving responses tailored to the intent of a threat initiator. We assess such threats against LLM-integrated applications empowered by OpenAI GPT-3.5 and GPT-4. Our empirical results show that the threats can effectively bypass the restrictions and moderation policies of OpenAI, resulting in users receiving responses that contain bias, toxic content, privacy risk, and disinformation. To mitigate those threats, we identify and define four key properties, namely integrity, source identification, attack detectability, and utility preservation, that need to be satisfied by a safe LLM-integrated application. Based on these properties, we develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.
Code (0)
등록된 구현이 없습니다.
Tasks
Code CompletionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities
Spurred by the recent rapid increase in the development and distribution of large language models (LLMs) across industry and academia, much recent work has drawn attention to safety- and security-related threats and vuln…
Time-Efficient Evaluation and Enhancement of Adversarial Robustness in Deep Neural Networks
With deep neural networks (DNNs) increasingly embedded in modern society, ensuring their safety has become a critical and urgent issue. In response, substantial efforts have been dedicated to the red-blue adversarial fra…
Adversarial RobustnessSEPTIC (Self Protecting Database): Detecting Injection Attacks and Vulnerabilities Inside the DBMS
Databases continue to be the most commonly used backend storage in enterprises, but they are often integrated with vulnerable applications, such as web frontends, which allow injection attacks to be performed. The effe…
3D Guard-Layer: An Integrated Agentic AI Safety System for Edge Artificial Intelligence
AI systems have found a wide range of real-world applications in recent years. The adoption of edge artificial intelligence, embedding AI directly into edge devices, is rapidly growing. Despite the implementation of guar…
BreachSeek: A Multi-Agent Automated Penetration Tester
The increasing complexity and scale of modern digital environments have exposed significant gaps in traditional cybersecurity penetration testing methods, which are often time-consuming, labor-intensive, and unable to ra…