Learning to Encode and Classify Test Executions
The challenge of automatically determining the correctness of test executions is referred to as the test oracle problem and is one of the key remaining issues for automated testing. The goal in this paper is to solve the test oracle problem in a way that is general, scalable and accurate. To achieve this, we use supervised learning over test execution traces. We label a small fraction of the execution traces with their verdict of pass or fail. We use the labelled traces to train a neural network (NN) model to learn to distinguish runtime patterns for passing versus failing executions for a given program. Our approach for building this NN model involves the following steps, 1. Instrument the program to record execution traces as sequences of method invocations and global state, 2. Label a small fraction of the execution traces with their verdicts, 3. Designing a NN component that embeds information in execution traces to fixed length vectors, 4. Design a NN model that uses the trace information for classification, 5. Evaluate the inferred classification model on unseen execution traces from the program. We evaluate our approach using case studies from different application domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Value pointer library and 6. Nine network protocols from Linux packet identifier, L7-Filter. We found the classification models for all subject programs resulted in high precision, recall and specificity, over 95%, while only training with an average 9% of the total traces. Our experiments show that the proposed neural network model is highly effective as a test oracle and is able to learn runtime patterns to distinguish passing and failing test executions for systems and tests from different application domains.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationSpecificityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Go-Oracle: Automated Test Oracle for Go Concurrency Bugs
The Go programming language has gained significant traction for developing software, especially in various infrastructure systems. Nonetheless, concurrency bugs have become a prevalent issue within Go, presenting a uniqu…
Using Metamorphic Relations to Verify and Enhance Artcode Classification
Software testing is often hindered where it is impossible or impractical to determine the correctness of the behaviour or output of the software under test (SUT), a situation known as the oracle problem. An example of an…
Classificationimage-classificationImage Classificationsoftware testingQuantum Observables for continuous control of the Quantum Approximate Optimization Algorithm via Reinforcement Learning
We present a classical control mechanism for Quantum devices using Reinforcement Learning. Our strategy is applied to the Quantum Approximate Optimization Algorithm (QAOA) in order to optimize an objective function that …
continuous-controlContinuous ControlQ-LearningReinforcement LearningAdaStop: adaptive statistical testing for sound comparisons of Deep RL agents
Recently, the scientific community has questioned the statistical reproducibility of many empirical results, especially in the field of machine learning. To contribute to the resolution of this reproducibility crisis, we…
Deep Reinforcement LearningMuJoCoReinforcement Learning (RL)A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder
The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-d…
Anomaly Detection