ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
Autonomous agents are increasingly expected to support scientific research, and recent benchmarks report progress in code repair and autonomous experimentation. However, these evaluations typically assume a pre-configured execution environment, which requires resolving complex software dependencies, aligning hardware and framework versions, and configuring distributed execution, yet this capability remains largely unbenchmarked. We introduce ResearchEnvBench, a benchmark for environment synthesis in research code execution. Given a research repository, documentation, and a target execution setting, agents must construct an environment that successfully executes at runtime. Evaluations on diverse research repositories reveal a substantial gap in current SOTA agents, with failures dominated by incomplete dependency resolution and brittle version coupling. ResearchEnvBench provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.
Code (0)
등록된 구현이 없습니다.
Tasks
Code RepairSimilar Papers 제목 키워드 기반
E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to…
Towards a Playground to Democratize Experimentation and Benchmarking of AI Agents for Network Troubleshooting
Recent research has demonstrated the effectiveness of Artificial Intelligence (AI), and more specifically, Large Language Models (LLMs), in supporting network configuration synthesis and automating network diagnosis task…
Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve ind…
Reinforcement LearningNovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds
As AI agents leave the lab and venture into the real world as autonomous vehicles, delivery robots, and cooking robots, it is increasingly necessary to design and comprehensively evaluate algorithms that tackle the ``ope…
Autonomous VehiclesBenchmarkingReinforcement Learning (RL)Benchmarking the Spectrum of Agent Capabilities
Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive tra…
Benchmarking