Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases

2026-03-24 · Dubai Li, Yuxiang He, Yan Hu, Yu Tian, Jingsong Li arxiv

Observational studies can yield clinically actionable evidence at scale, but executing them on real-world databases is open-ended and requires coherent decisions across cohort construction, analysis, and reporting. Prior evaluations of LLM agents emphasize isolated steps or single answers, missing the integrity and internal structure of the resulting evidence bundle. To address this gap, we introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies. Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database and iteratively generate tree-structured evidence bundles. We evaluate six LLMs (three open-source, three closed-source) under three agent scaffolds using both question-level correctness and end-to-end task metrics. Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%. Agent scaffolds also matter substantially, causing over 30% variation in performance metrics. Furthermore, we implement an automated cohort evaluation method to rapidly localize errors and identify agent failure modes. Overall, the results highlight persistent limitations in agents' ability to produce end-to-end evidence bundles, and efficient validation remains an important direction for future work. Code and data are available at https://github.com/somewordstoolate/RWE-bench.

📄 PDF Abstract BibTeX arXiv:2603.22767

Code (0)

등록된 구현이 없습니다.

WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning

2026-06-10 · Yizhou Chi, Eric Chamoun, Zifeng Ding, Andreas Vlachos arxiv

Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether agents genuinely forecast requires more than final-answer accurac…

Evaluating the Search Agent in a Parallel World

2026-03-05 · Jiawei Chen, Xintian Shen, Lihao Zheng, Lifu Mu 외 arxiv

Integrating web search tools has significantly extended the capability of LLMs to address open-world, real-time, and long-tail problems. However, evaluating these Search Agents presents formidable challenges. First, cons…

BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research

2025-05-22 · Zifeng Wang, Benjamin Danek, Jimeng Sun

Validating scientific hypotheses is a central challenge in biomedical research, and remains difficult for artificial intelligence (AI) agents due to the complexity of real-world data analysis and evidence interpretation.…

Benchmarking

Dr-CiK: A Testbed for Foresight-Driven Agents

2026-05-27 · Yihong Tang, Andrew Robert Williams, Arjun Ashok, Vincent Zhihao Zheng 외 arxiv

Time series forecasting in real-world settings often depends not only on historical observations, but also on external context that must be actively discovered from noisy, heterogeneous information sources. Yet existing …

Time Series Forecasting

Do LLM Agents Know How to Ground, Recover, and Assess? A Benchmark for Epistemic Competence in Information-Seeking Agents

2025-09-26 · Jiaqi Shao, Yuxiang Lin, Munish Prasad Lohani, Yufeng Miao 외 arxiv

Recent work has explored training Large Language Model (LLM) search agents with reinforcement learning (RL) for open-domain question answering (QA). However, most evaluations focus solely on final answer accuracy, overlo…

Open-Domain Question Answering Reinforcement Learning