Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces Retecs, a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The Retecs method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the Retecs method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying Retecs on data extracted from three industrial case studies, we show for the first time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing.
Code (2)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
XCS as a reinforcement learning approach to automatic test case prioritization
Testing is a crucial part in the development of new products. With the rise of test automation methods, companies start relying on an even higher number of tests. Sometimes it is not feasible to run all tests and the goa…
reinforcement-learningReinforcement Learning (RL)XCSF for Automatic Test Case Prioritization
Testing is a crucial part in the development of a new product. Due to the change from manual testing to automated testing, companies can rely on a higher number of tests. There are certain cases such as smoke tests whe…
reinforcement-learningReinforcement Learning (RL)XCSF with Experience Replay for Automatic Test Case Prioritization
The verification of a new product is of major importance for companies. With the rise of test automation, companies start to rely on huge numbers of tests. Often, it is not feasible to run all available tests due to time…
PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization
Mutation testing is a widely recognized technique for assessing and enhancing the effectiveness of software test suites by introducing deliberate code mutations. However, its application often results in overly large tes…
Machine Learning Techniques for Software Quality Assurance: A Survey
Over the last years, machine learning techniques have been applied to more and more application domains, including software engineering and, especially, software quality assurance. Important application domains have been…
BIG-bench Machine LearningFault DetectionPredictionSurvey