PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities. Our code is available at https://github.com/kilian-group/phantom-wiki.
Code (1)
Tasks
RetrievalSimilar Papers 제목 키워드 기반
Memento: Note-Taking for Your Future Self
Large language models (LLMs) excel at reasoning-only tasks, but struggle when reasoning must be tightly coupled with retrieval, as in multi-hop question answering. To overcome these limitations, we introduce a prompting …
Multi-hop Question AnsweringQuestion AnsweringRAGMR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval
Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios. Existing benchmarks primarily probe surface-level se…
Semantic correspondenceCausal InferenceCoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method
Legal consultation questions exhibit multi-level complexity. A single retrieval strategy often leads to over-reasoning for simple questions and poor interpretability for complex ones, making it difficult to meet the requ…
Semantic RetrievalLogical ReasoningNoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding
Understanding and reasoning over academic handwritten notes remains a challenge in document AI, particularly for mathematical equations, diagrams, and scientific notations. Existing visual question answering (VQA) benchm…
BenchmarkingDocument AIdocument understandingMultimodal Reasoning+6MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more soph…
Multimodal Reasoning