Is AUC the best measure for practical comparison of anomaly detectors?
The area under receiver operating characteristics (AUC) is the standard measure for comparison of anomaly detectors. Its advantage is in providing a scalar number that allows a natural ordering and is independent on a threshold, which allows to postpone the choice. In this work, we question whether AUC is a good metric for anomaly detection, or if it gives a false sense of comfort, due to relying on assumptions which are unlikely to hold in practice. Our investigation shows that variations of AUC emphasizing accuracy at low false positive rate seem to be better correlated with the needs of practitioners, but also that we can compare anomaly detectors only in the case when we have representative examples of anomalous samples. This last result is disturbing, as it suggests that in many cases, we should do active or few-show learning instead of pure anomaly detection.
Code (1)
Tasks
Anomaly DetectionSimilar Papers 제목 키워드 기반
Evaluating Anomaly Detectors for Simulated Highly Imbalanced Industrial Classification Problems
Machine learning offers potential solutions to current issues in industrial systems in areas such as quality control and predictive maintenance, but also faces unique barriers in industrial applications. An ongoing chall…
Anomaly DetectionTraining-Free Logical and Structural Anomaly Detection via Calibrated Fusion
Industrial anomaly detection must handle two distinct defect families: structural anomalies, which manifest as local texture corruptions, and logical anomalies, which violate global rules on object count, composition, or…
Anomaly DetectionComparison of Anomaly Detectors: Context Matters
Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objecti…
Anomaly DetectionModel SelectionExperience Report: Deep Learning-based System Log Analysis for Anomaly Detection
Logs have been an imperative resource to ensure the reliability and continuity of many software systems, especially large-scale distributed systems. They faithfully record runtime information to facilitate system trouble…
Anomaly DetectionECoLAD: Selecting Anomaly Detectors for Automotive Deployment via Compute-Reduction Evaluation
Automotive anomaly detectors are often selected from accuracy only benchmarks on workstation class hardware, whereas in-vehicle monitoring requires predictable scoring latency under limited CPU parallelism. This mismatch…
Anomaly Detection