UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

2024-06-21 · Yulong Hui, Yao Lu, Huanchen Zhang

The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.

📄 PDF Abstract BibTeX arXiv:2406.15187

Code (1)

qinchuanhui/uda-benchmark 공식 구현

Tasks

Question Answering RAG RetrievalRetrieval-augmented Generation

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

2026-05-20 · Junhao Ruan, Abudukeyumu Abudula, Bei Li, Yongjing Yin 외 arxiv

Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or …

MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation

2025-09-29 · Yuelyu Ji, Wuwei Lan, Patrick NG arxiv

Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty an…

Question Answering

WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation

2025-05-13 · Dvir Cohen, Lin Burg, Sviatoslav Pykhnivskyi, Hagit Gur 외

Retrieval-Augmented Generation (RAG) is a cornerstone of modern question answering (QA) systems, enabling grounded answers based on external knowledge. Although recent progress has been driven by open-domain datasets, en…

Question Answering RAG Retrieval Retrieval-augmented Generation

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

2026-05-19 · Gioele Molinari, Florian Felten, Soheyl Massoudi, Mark Fuge arxiv

Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturi…

Instruction Following

ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry

2025-06-26 · Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi 외

Community Question Answering (CQA) platforms can be deemed as important knowledge bases in community, but effectively leveraging historical interactions and domain knowledge in real-time remains a challenge. Existing met…

Community Question Answering Question Answering Retrieval Retrieval-augmented Generation