Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

2025-04-21 · Aoran Gan, Hao Yu, Kai Zhang, Qi Liu, Wenyu Yan, Zhenya Huang, Shiwei Tong, Guoping Hu

Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual accuracy, safety, and computational efficiency in the LLM era. We also compile and categorize the RAG-specific datasets and evaluation frameworks, conducting a meta-analysis of evaluation practices in high-impact RAG research. To the best of our knowledge, this work represents the most comprehensive survey for RAG evaluation, bridging traditional and LLM-driven methods, and serves as a critical resource for advancing RAG development.

📄 PDF Abstract BibTeX arXiv:2504.14891

Code (1)

yhpeter/awesome-rag-evaluation

Tasks

Computational Efficiency Information Retrieval RAG RetrievalRetrieval-augmented GenerationSurveyText Generation

Methods 이 논문이 사용한 방법론

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Benchmarking Large Language Models in Retrieval-Augmented Generation

2023-09-04 · Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented gen…

Benchmarking counterfactual Hallucination RAG+2

Controlled Retrieval-augmented Context Evaluation for Long-form RAG

2025-06-24 · Jia-Huei Ju, Suzan Verberne, Maarten de Rijke, Andrew Yates

Retrieval-augmented generation (RAG) enhances large language models by incorporating context retrieved from external knowledge sources. While the effectiveness of the retrieval module is typically evaluated with relevanc…

Diagnostic Form RAG Retrieval+1

From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL

2023-11-11 · Xiaoqian Li, Ercong Nie, Sheng Liang

The remarkable ability of Large Language Models (LLMs) to understand and follow instructions has sometimes been limited by their in-context learning (ICL) performance in low-resource languages. To address this, we introd…

In-Context Learning Retrieval

Meta-prompting Optimized Retrieval-augmented Generation

2024-07-04 · João Rodrigues, António Branco

Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible…

Multi-hop Question Answering Question Answering Retrieval Retrieval-augmented Generation+1

Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)

2026-05-14 · Assaf Gerner, Netta Madvil, Nadav Barak, Alex Zaikman 외 arxiv

Large Language Models (LLMs) augmented with Retrieval-Augmented Generation (RAG) techniques are revolutionizing applications across multiple domains, such as healthcare, finance, and customer service. Despite their poten…