LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops
Large Language Models (LLMs) are showing remarkable performance in generating source code, yet the generated code often has issues like compilation errors or incorrect code. Researchers and developers often face wasted effort in implementing checks and refining LLM-generated code, frequently duplicating their efforts. This paper presents LLMLOOP, a framework that automates the refinement of both source code and test cases produced by LLMs. LLMLOOP employs five iterative loops: resolving compilation errors, addressing static analysis issues, fixing test case failures, and improving test quality through mutation analysis. These loops ensure the generation of high-quality test cases that serve as both a validation mechanism and a regression test suite for the generated code. We evaluated LLMLOOP on HUMANEVAL-X, a recent benchmark of programming tasks. Results demonstrate the tool's effectiveness in refining LLM-generated outputs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Provenance and Pseudo-Provenance for Seeded Learning-Based Automated Test Generation
Many methods for automated software test generation, including some that explicitly use machine learning (and some that use ML more broadly conceived) derive new tests from existing tests (often referred to as seeds). Of…
LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language M…
Prompt EngineeringA System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites
Unit tests represent the most basic level of testing within the software testing lifecycle and are crucial to ensuring software correctness. Designing and creating unit tests is a costly and labor-intensive process that …
Code Generationsoftware testingBACE: LLM-based Code Generation through Bayesian Anchored Co-Evolution of Code and Test Populations
Large Language Models (LLMs) have demonstrated impressive capabilities in code generation. While an interactive feedback loop can improve performance, writing effective tests is a non-trivial task. Early multi-agent fram…
Code GenerationHarden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts rais…
software testing