ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection
With the increasing prevalence of scalable file systems in the context of High Performance Computing (HPC), the importance of accurate anomaly detection on runtime logs is increasing. But as it currently stands, many state-of-the-art methods for log-based anomaly detection, such as DeepLog, have encountered numerous challenges when applied to logs from many parallel file systems (PFSes), often due to their irregularity and ambiguity in time-based log sequences. To circumvent these problems, this study proposes ClusterLog, a log pre-processing method that clusters the temporal sequence of log keys based on their semantic similarity. By grouping semantically and sentimentally similar logs, this approach aims to represent log sequences with the smallest amount of unique log keys, intending to improve the ability of a downstream sequence-based model to effectively learn the log patterns. The preliminary results of ClusterLog indicate not only its effectiveness in reducing the granularity of log sequences without the loss of important sequence information but also its generalizability to different file systems' logs.
Code (0)
등록된 구현이 없습니다.
Tasks
Anomaly DetectionClusteringSemantic SimilaritySemantic Textual SimilaritySimilar Papers 제목 키워드 기반
A Supervised Embedding and Clustering Anomaly Detection method for classification of Mobile Network Faults
The paper introduces Supervised Embedding and Clustering Anomaly Detection (SEMC-AD), a method designed to efficiently identify faulty alarm logs in a mobile network and alleviate the challenges of manual monitoring caus…
Anomaly DetectionClusteringimbalanced classificationLogSHIELD: A Graph-based Real-time Anomaly Detection Framework using Frequency Analysis
Anomaly-based cyber threat detection using deep learning is on a constant growth in popularity for novel cyber-attack detection and forensics. A robust, efficient, and real-time threat detector in a large-scale operation…
Anomaly DetectionCyber Attack DetectionGraph EmbeddingGraph Neural NetworkGood Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels
Anomaly detection often relies on supervised or clustering approaches, with limited success in specialized domains like automotive communication systems where scalable solutions are essential. We propose a novel decoder-…
Anomaly DetectionAnomaly Detection for Network Connection Logs
We leverage a streaming architecture based on ELK, Spark and Hadoop in order to collect, store, and analyse database connection logs in near real-time. The proposed system investigates outliers using unsupervised learnin…
Anomaly DetectionClusteringGeneral ClassificationDetecting Log Anomalies with Multi-Head Attention (LAMA)
Anomaly detection is a crucial and challenging subject that has been studied within diverse research areas. In this work, we explore the task of log anomaly detection (especially computer system logs and user behavior lo…
Anomaly Detection