Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark
Much of the world's data is streaming, time-series data, where anomalies give significant information in critical situations; examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. There are no benchmarks to adequately test and score the efficacy of real-time anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which attempts to provide a controlled and repeatable environment of open-source tools to test and measure anomaly detection algorithms on streaming data. The perfect detector would detect all anomalies as soon as possible, trigger no false alarms, work with real-world time-series data across a variety of domains, and automatically adapt to changing statistics. Rewarding these characteristics is formalized in NAB, using a scoring algorithm designed for streaming data. NAB evaluates detectors on a benchmark dataset with labeled, real-world time-series data. We present these components, and give results and analyses for several open source, commercially-used algorithms. The goal for NAB is to provide a standard, open source framework with which the research community can compare and evaluate different algorithms for detecting anomalies in streaming data.
Code (3)
Tasks
Anomaly DetectionTime SeriesTime Series AnalysisSimilar Papers 제목 키워드 기반
Texture-AD: An Anomaly Detection Dataset and Benchmark for Real Algorithm Development
Anomaly detection is a crucial process in industrial manufacturing and has made significant advancements recently. However, there is a large variance between the data used in the development and the data collected by the…
Anomaly DetectionDefect DetectionUnsupervised Anomaly DetectionPATE: Proximity-Aware Time series anomaly Evaluation
Evaluating anomaly detection algorithms in time series data is critical as inaccuracies can lead to flawed decision-making in various domains where real-time analytics and data-driven strategies are essential. Traditiona…
Anomaly DetectionDecision MakingTime SeriesASBench: Image Anomalies Synthesis Benchmark for Anomaly Detection
Anomaly detection plays a pivotal role in manufacturing quality control, yet its application is constrained by limited abnormal samples and high manual annotation costs. While anomaly synthesis offers a promising solutio…
Anomaly DetectionA Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data
Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model and assign anoma…
Anomaly DetectionTime SeriesUnsupervised Anomaly DetectionAn Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series
Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This paper presents a systematic and comprehe…
Anomaly DetectionTime SeriesTime Series AnalysisTime Series Anomaly Detection