HeurekaBench: A Benchmarking Framework for AI Co-scientist
LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Scientific Machine Learning Benchmarks
The breakthrough in Deep Learning neural networks has transformed the use of AI and machine learning technologies for the analysis of very large experimental datasets. These datasets are typically generated by large-scal…
BenchmarkingBIG-bench Machine LearningAdversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities
Benchmarking the ability of AI scientists to generate novel ideas is notoriously difficult. Existing benchmarks in this field have made progress in evaluating scientific reasoning and research replication, but often rely…
Python Random Graph Generator
Pyrgg is an easy-to-use synthetic random graph generator written in Python which supports various graph file formats including DIMACS .gr files. Pyrgg has the ability to generate graphs of different sizes and is designed…
BenchmarkingGraph GenerationBenchmarking AI scientists in omics data-driven biological research
The rise of large language models and multi-agent systems has sparked growing interest in AI scientists capable of autonomous biological research. However, existing benchmarks either focus on reasoning without data or on…
BenchmarkingMultiple-choicescientific discoveryCausalGame: Benchmarking Causal Thinking of LLM Agents in Games
Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capabilit…