LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.
Code (0)
등록된 구현이 없습니다.
Tasks
Code CompletionCode GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift …
Code GenerationCorrect Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
Large language models (LLMs) are now largely involved in software development workflows, and the code they generate routinely includes third-party library (TPL) imports annotated with specific version identifiers. These …
GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward compatibility. While existing code evol…
Code GenerationCode CompletionSVInvNet: A Densely Connected Encoder-Decoder Architecture for Seismic Velocity Inversion
This study presents a deep learning-based approach to seismic velocity inversion problem, focusing on both noisy and noiseless training datasets of varying sizes. Our Seismic Velocity Inversion Network (SVInvNet) introdu…
DecoderAutomating Code Adaptation for MLOps -- A Benchmarking Study on LLMs
This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of O…
BenchmarkingHyperparameter OptimizationModel OptimizationTranslation