Metamorphic Testing of a Deep Learning based Forecaster
In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.
Code (1)
Tasks
Deep LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Object-based Metamorphic Testing through Image Structuring
Testing software is often costly due to the need of mass-producing test cases and providing a test oracle for it. This is often referred to as the oracle problem. One method that has been proposed in order to alleviate t…
ObjectEvaluating Human Trajectory Prediction with Metamorphic Testing
The prediction of human trajectories is important for planning in autonomous systems that act in the real world, e.g. automated driving or mobile robots. Human trajectory prediction is a noisy process, and no prediction …
PredictionTrajectory PredictionFrom Untestable to Testable: Metamorphic Testing in the Age of LLMs
This article discusses the challenges of testing software systems with increasingly integrated AI and LLM functionalities. LLMs are powerful but unreliable, and labeled ground truth for testing rarely scales. Metamorphic…
MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems
With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in …
MTGP: Combining Metamorphic Testing and Genetic Programming
Genetic programming is an evolutionary approach known for its performance in program synthesis. However, it is not yet mature enough for a practical use in real-world software development, since usually many training cas…
Program Synthesis