A recommender system for efficient discovery of new anomalies in large-scale access logs
We present a novel, non-standard recommender system for large-scale security policy management(SPM). Our system Helios discovers and recommends unknown and unseen anomalies in large-scale access logs with minimal supervision and no starting information on users and items. Typical recommender systems assume availability of user- and item-related information, but such information is not usually available in access logs. To resolve this problem, we first use discrete categorical labels to construct categorical combinations from access logs in a bootstrapping manner. Then, we utilize rank statistics of entity rank and order categorical combinations for recommendation. From a double-sided cold start, with minimal supervision, Helios learns to recommend most salient anomalies at large-scale, and provides visualizations to security experts to explain rationale behind the recommendations. Our experiments show Helios to be suitable for large-scale applications: from cold starts, in less than 60 minutes, Helios can analyze roughly 4.6 billion records in logs of 400GB with about 300 million potential categorical combinations, then generate ranked categorical combinations as recommended discoveries. We also show that, even with limited computing resources, Helios accelerates unknown and unseen anomaly discovery process for SPM by 1 to 3 orders of magnitude, depending on use cases. In addition, Helios' design is flexible with metrics and measurement fields used for discoveries and recommendations. Overall, our system leads to more efficient and customizable SPM processes with faster discoveries of unseen and unknown anomalies.
Code (0)
등록된 구현이 없습니다.
Tasks
ManagementRecommendation SystemsSimilar Papers 제목 키워드 기반
Deploying Semantic ID-based Generative Retrieval for Large-Scale Podcast Discovery at Spotify
Podcast listening is often grounded in a set of favorite shows, while listener intent can evolve over time. This combination of stable preferences and changing intent motivates recommendation approaches that support both…
CAM2: Conformity-Aware Multi-Task Ranking Model for Large-Scale Recommender Systems
Learning large-scale industrial recommender system models by fitting them to historical user interaction data makes them vulnerable to conformity bias. This may be due to a number of factors, including the fact that user…
Recommendation SystemsPIE: Personalized Interest Exploration for Large-Scale Recommender Systems
Recommender systems are increasingly successful in recommending personalized content to users. However, these systems often capitalize on popular content. There is also a continuous evolution of user interests that need …
Recommendation SystemsExplainable Online Unsupervised Anomaly Detection for Cyber-Physical Systems via Causal Discovery from Time Series
Online unsupervised detection of anomalies is crucial to guarantee the correct operation of cyber-physical systems and the safety of humans interacting with them. State-of-the-art approaches based on deep learning via ne…
Anomaly DetectionCausal DiscoveryTime SeriesUnsupervised Anomaly DetectionPolyRecommender: A Multimodal Recommendation System for Polymer Discovery
We introduce PolyRecommender, a multimodal discovery framework that integrates chemical language representations from PolyBERT with molecular graph-based representations from a graph encoder. The system first retrieves c…
Multimodal Recommendation