TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting
Long-term time-series forecasting remains challenging due to the difficulty in capturing long-term dependencies, achieving linear scalability, and maintaining computational efficiency. We introduce TimeMachine, an innova…
Computational EfficiencyMambaTime SeriesTime Series ForecastingFEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we intr…
Code CompletionCode GenerationNEWTON: Are Large Language Models Capable of Physical Reasoning?
Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited explora…
AttributeCommon Sense ReasoningCoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark
As Large Language Models (LLMs) demonstrate increasingly sophisticated code processing capabilities, evaluating their performance on engineering-level code remains challenging. Existing repository-level benchmarks primar…
Bug fixingCode Generationtest driven developmentTurning the Tide: Repository-based Code Reflection
Code large language models (LLMs) enhance programming by understanding and generating code across languages, offering intelligent feedback, bug detection, and code updates through reflection, improving development effici…
Code GenerationDiversityHumanEval