EnvBench

홈페이지 · 논문 1편

EnvBench is a comprehensive benchmark for automating environment setup - an important task in software engineering. We have collected the largest dataset to date for this task and introduced a robust framework for developing and evaluating LLM-based agents that tackle environment setup challenges. Our benchmark includes: * 994 repositories: 329 Python and 665 JVM-based (Java, Kotlin) projects * Genuine configuration challenges: Carefully selected repositories that cannot be configured with simple deterministic scripts * Evaluation metrics: Static analysis for missing imports in Python and compilation checks for JVM languages * Baselines: Zero-shot baselines and agentic workflows tested with GPT-4o and GPT-4o-mini