홈 › Papers

SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model

2025-01-28 · Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang

The indexing-retrieval-generation paradigm of retrieval-augmented generation (RAG) has been highly successful in solving knowledge-intensive tasks by integrating external knowledge into large language models (LLMs). However, the incorporation of external and unverified knowledge increases the vulnerability of LLMs because attackers can perform attack tasks by manipulating knowledge. In this paper, we introduce a benchmark named SafeRAG designed to evaluate the RAG security. First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service. Next, we construct RAG security evaluation dataset (i.e., SafeRAG dataset) primarily manually for each task. We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter. Experiments conducted on 14 representative RAG components demonstrate that RAG exhibits significant vulnerability to all attack tasks and even the most apparent attack task can easily bypass existing retrievers, filters, or advanced LLMs, resulting in the degradation of RAG service quality. Code is available at: https://github.com/IAAR-Shanghai/SafeRAG.

📄 PDF Abstract BibTeX arXiv:2501.18636

Code (1)

iaar-shanghai/saferag 공식 구현

Tasks

BenchmarkingLanguage ModelingLanguage ModellingLarge Language ModelRAG RetrievalRetrieval-augmented Generation

Methods 이 논문이 사용한 방법론

Golden Queue Managers 설명 없음

Attention 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

WordPiece 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

EcoSafeRAG: Efficient Security through Context Analysis in Retrieval-Augmented Generation

2025-05-16 · Ruobing Yao, Yifei Zhang, Shuang Song, Neng Gao 외

Retrieval-Augmented Generation (RAG) compensates for the static knowledge limitations of Large Language Models (LLMs) by integrating external knowledge, producing responses with enhanced factual correctness and query-spe…

Diversity RAG Retrieval Retrieval-augmented Generation+2

Steering Over-refusals Towards Safety in Retrieval Augmented Generation

2025-10-12 · Utsav Maskey, Mark Dras, Usman Naseem arxiv

Safety alignment in large language models (LLMs) induces over-refusals -- where LLMs decline benign requests due to aggressive safety filters. We analyze this phenomenon in retrieval-augmented generation (RAG), where bot…

Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generation

2025-10-31 · Arnabh Borah, Md Tanvirul Alam, Nidhi Rastogi arxiv

Security applications are increasingly relying on large language models (LLMs) for cyber threat detection; however, their opaque reasoning often limits trust, particularly in decisions that require domain-specific cybers…

A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

2025-07-25 · Agada Joseph Oche, Ademola Glory Folashade, Tirthankar Ghosal, Arpan Biswas arxiv

Retrieval-Augmented Generation (RAG) represents a major advancement in natural language processing (NLP), combining large language models (LLMs) with information retrieval systems to enhance factual grounding, accuracy, …

Computational Efficiency Information Retrieval Question Answering

Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback

2026-01-01 · Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj 외 arxiv

Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors. Prior work demonstrates that LLMs benefit substantially from structured feedba…

Semantic Retrieval Code Generation