ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation

2026-03-27 · Xianpeng, Sun, Haonan Sun, Tian Yu, Sheng Ma, Qincheng Zhang, Lifei Rao, Chen Tian arxiv

Evaluation of repository-aware software engineering systems is often confounded by synthetic task design, prompt leakage, and temporal contamination between repository knowledge and future code changes. We present a time-consistent benchmark methodology that snapshots a repository at time T0, constructs repository-derived code knowledge using only artifacts available before T0, and evaluates on engineering tasks derived from pull requests merged in the future interval (T0, T1]. Each historical pull request is transformed into a natural-language task through an LLM-assisted prompt-generation pipeline, and the benchmark is formalized as a matched A/B comparison in which the same software engineering agent is evaluated with and without repository-derived code knowledge while all other variables are held constant. We also report a baseline characterization study on two open-source repositories, DragonFly and React, using three Claude-family models and four prompt granularities. Across both repositories, file-level F1 increases monotonically from minimal to guided prompts, reaching 0.8081 on DragonFly and 0.8078 on React for the strongest tested model. These results show that prompt construction is a first-order benchmark variable. More broadly, the benchmark highlights that temporal consistency and prompt control are core validity requirements for repository-aware software engineering evaluation.

📄 PDF Abstract BibTeX arXiv:2603.26137

Code (0)

등록된 구현이 없습니다.

RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph

2024-10-03 · Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao 외

Large Language Models (LLMs) excel in code generation yet struggle with modern AI software engineering tasks. Unlike traditional function-level or file-level coding tasks, AI software engineering requires not only basic …

Code Generation

ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data

2022-03-15 · Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo 외

Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their …

Benchmarking Domain Adaptation Time Series Time Series Analysis+1

Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development

2026-04-08 · Xinchen Wang, Ruida Hu, Cuiyun Gao, Pengfei Gao 외 arxiv

Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, existing benchmarks have two key limitations:…

Question Answering

Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling

2025-07-31 · Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng 외 arxiv

Software issue resolution is a critical challenge in software engineering and has garnered increasing attention in recent years. With the rapid advancement of large language models (LLMs), substantial progress has been m…

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

2026-03-26 · Jiseung Hong, Benjamin G. Ascoli, Jinho D. Choi arxiv

Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range o…