Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects
Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic and simple smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that considerations how they can be adapted to this new context can help to find and prevent severe bugs, even in mature machine learning libraries.
Code (1)
Tasks
BIG-bench Machine Learningsoftware testingvalidSimilar Papers 제목 키워드 기반
XCSF for Automatic Test Case Prioritization
Testing is a crucial part in the development of a new product. Due to the change from manual testing to automated testing, companies can rely on a higher number of tests. There are certain cases such as smoke tests whe…
reinforcement-learningReinforcement Learning (RL)AutoML Two-Sample Test
Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going …
AutoMLscientific discoveryTwo-sample testingVocal Bursts Valence PredictionUsing Machine Learning to Develop Smart Reflex Testing Protocols
Objective: Reflex testing protocols allow clinical laboratories to perform second line diagnostic tests on existing specimens based on the results of initially ordered tests. Reflex testing can support optimal clinical l…
DiagnosticImputationManagementPAPRIKA: Private Online False Discovery Rate Control
In hypothesis testing, a false discovery occurs when a hypothesis is incorrectly rejected due to noise in the sample. When adaptively testing multiple hypotheses, the probability of a false discovery increases as more te…
Two-sample testingTransmission-Guided Bayesian Generative Model for Smoke Segmentation
Smoke segmentation is essential to precisely localize wildfire so that it can be extinguished in an early phase. Although deep neural networks have achieved promising results on image segmentation tasks, they are prone t…
Image DehazingImage SegmentationmodelSegmentation+1