RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

2023-08-21 · Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka, Osamu Torii, Jun Deguchi

Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks. We open-source our code at https://github.com/yhoshi3/RaLLe.

📄 PDF Abstract BibTeX arXiv:2308.10633

Code (1)

yhoshi3/ralle 공식 구현

Tasks

Information Retrieval Question Answering Retrieval

Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking

2026-05-28 · Songbo Hu, Yinhong Liu, Ej Zhou, Evgeniia Razumovskaia 외 arxiv

Creating spoken dialogue datasets is methodologically challenging, and these challenges are amplified when the goal is to build multilingual, multi-parallel datasets at scale. This work introduces HEALTHDIAL, a large-sca…

Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)

2026-05-14 · Assaf Gerner, Netta Madvil, Nadav Barak, Alex Zaikman 외 arxiv

Large Language Models (LLMs) augmented with Retrieval-Augmented Generation (RAG) techniques are revolutionizing applications across multiple domains, such as healthcare, finance, and customer service. Despite their poten…

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

2026-07-02 · Valentin J. J. Kreileder, Johannes Reisinger, Andreas Fischer arxiv

Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves…

Evaluating Cultural Knowledge Processing in Large Language Models: A Cognitive Benchmarking Framework Integrating Retrieval-Augmented Generation

2025-11-03 · Hung-Shin Lee, Chen-Chi Chang, Ching-Yuan Chen, Yun-Hsiang Hsu arxiv

This study proposes a cognitive benchmarking framework to evaluate how large language models (LLMs) process and apply culturally specific knowledge. The framework integrates Bloom's Taxonomy with Retrieval-Augmented Gene…

EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation

2024-12-17 · Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song 외

We introduce EXIT, an extractive context compression framework that enhances both the effectiveness and efficiency of retrieval-augmented generation (RAG) in question answering (QA). Current RAG systems often struggle wh…

Question Answering RAG Retrieval Retrieval-augmented Generation+1