Log-based Anomaly Detection based on EVT Theory with feedback
System logs play a critical role in maintaining the reliability of software systems. Fruitful studies have explored automatic log-based anomaly detection and achieved notable accuracy on benchmark datasets. However, when applied to large-scale cloud systems, these solutions face limitations due to high resource consumption and lack of adaptability to evolving logs. In this paper, we present an accurate, lightweight, and adaptive log-based anomaly detection framework, referred to as SeaLog. Our method introduces a Trie-based Detection Agent (TDA) that employs a lightweight, dynamically-growing trie structure for real-time anomaly detection. To enhance TDA's accuracy in response to evolving log data, we enable it to receive feedback from experts. Interestingly, our findings suggest that contemporary large language models, such as ChatGPT, can provide feedback with a level of consistency comparable to human experts, which can potentially reduce manual verification efforts. We extensively evaluate SeaLog on two public datasets and an industrial dataset. The results show that SeaLog outperforms all baseline methods in terms of effectiveness, runs 2X to 10X faster and only consumes 5% to 41% of the memory resource.
Code (0)
등록된 구현이 없습니다.
Tasks
Anomaly DetectionSimilar Papers 제목 키워드 기반
GLAD: GLocalized Anomaly Detection via Human-in-the-Loop Learning
Human analysts that use anomaly detection systems in practice want to retain the use of simple and explainable global anomaly detectors. In this paper, we propose a novel human-in-the-loop learning algorithm called GLAD …
Anomaly DetectionDetecting Irregular Network Activity with Adversarial Learning and Expert Feedback
Anomaly detection is a ubiquitous and challenging task relevant across many disciplines. With the vital role communication networks play in our daily lives, the security of these networks is imperative for smooth functio…
Anomaly DetectionContrastive LearningDimensionality Increment of PMU Data for Anomaly Detection in Low Observability Power Systems
Anomaly detection is an important task in power systems. To make better use of the phasor measurement unit (PMU) data collected from a low observability power system for anomaly detection, a data dimensionality increment…
Anomaly DetectionUnsupervised Anomaly Detection Ensembles using Item Response Theory
Constructing an ensemble from a heterogeneous set of unsupervised anomaly detection methods is challenging because the class labels or the ground truth is unknown. Thus, traditional ensemble techniques that use the respo…
Anomaly DetectionUnsupervised Anomaly DetectionRethinking Autoencoders for Medical Anomaly Detection from A Theoretical Perspective
Medical anomaly detection aims to identify abnormal findings using only normal training data, playing a crucial role in health screening and recognizing rare diseases. Reconstruction-based methods, particularly those uti…
Anomaly DetectionPhilosophy