Towards a Rigorous Evaluation of Time-series Anomaly Detection
In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements in TAD. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; that is, even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods after applying the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even when PA is forbidden. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.
Code (1)
Tasks
Anomaly DetectionTime SeriesTime Series AnalysisTime Series Anomaly DetectionSimilar Papers 제목 키워드 기반
Unsupervised Anomaly Detection in Time-series: An Extensive Evaluation and Analysis of State-of-the-art Methods
Unsupervised anomaly detection in time-series has been extensively investigated in the literature. Notwithstanding the relevance of this topic in numerous application fields, a comprehensive and extensive evaluation of r…
Anomaly DetectionTime SeriesTime Series AnalysisTime Series Anomaly Detection+1Fun-TSG: A Function-Driven Multivariate Time Series Generator with Variable-Level Anomaly Labeling
Reliable evaluation of anomaly detection methods in multivariate time series remains an open challenge, largely due to the limitations of existing benchmark datasets. Current resources often lack fine-grained anomaly ann…
Anomaly DetectionPaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection
Although recent studies on time-series anomaly detection have increasingly adopted ever-larger neural network architectures such as transformers and foundation models, they incur high computational costs and memory usage…
Representation LearningAnomaly DetectionmTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale
Multivariate time series anomaly detection (MTS-AD) is critical in domains like healthcare, cybersecurity, and industrial monitoring, yet remains challenging due to complex inter-variable dependencies, temporal dynamics,…
Anomaly DetectionBenchmarkingLanguage ModelingLanguage Modelling+4MSAD: A Deep Dive into Model Selection for Time series Anomaly Detection
Anomaly detection is a fundamental task for time series analytics with important implications for the downstream performance of many applications. Despite increasing academic interest and the large number of methods prop…
Time Series Anomaly DetectionTime Series Classification