Accelerating System Log Processing by Semi-supervised Learning: A Technical Report
There is an increasing need for more automated system-log analysis tools for large scale online system in a timely manner. However, conventional way to monitor and classify the log output based on keyword list does not scale well for complex system in which codes contributed by a large group of developers, with diverse ways of encoding the error messages, often with misleading pre-set labels. In this paper, we propose that the design of a large scale online log analysis should follow the "Least Prior Knowledge Principle", in which unsupervised or semi-supervised solution with the minimal prior knowledge of the log should be encoded directly. Thereby, we report our experience in designing a two-stage machine learning based method, in which the system logs are regarded as the output of a quasi-natural language, pre-filtered by a perplexity score threshold, and then undergo a fine-grained classification procedure. Tests on empirical data show that our method has obvious advantage regarding to the processing speed and classification accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Automating the search for a patent's prior art with a full text similarity search
More than ever, technical inventions are the symbol of our society's advance. Patents guarantee their creators protection against infringement. For an invention being patentable, its novelty and inventiveness have to be …
text similaritySemi-supervised Network Embedding with Differentiable Deep Quantisation
Learning accurate low-dimensional embeddings for a network is a crucial task as it facilitates many downstream network analytics tasks. For large networks, the trained embeddings often require a significant amount of spa…
Link PredictionNetwork EmbeddingNode ClassificationPrediction+1DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use …
Depth EstimationDomain GeneralizationFine-Grained Image ClassificationImage Classification+5Unsupervised Generation of Long-form Technical Questions from Textbook Metadata using Structured Templates
We explore the task of generating long-form technical questions from textbooks. Semi-structured metadata of a textbook — the table of contents and the index — provide rich cues for technical question generation. Existing…
FormQuestion GenerationQuestion-GenerationReading ComprehensionSemi-Supervised Learning for Molecular Graphs via Ensemble Consensus
Machine learning is transforming molecular sciences by accelerating property prediction, simulation, and the discovery of new molecules and materials. Acquiring labeled data in these domains is often costly and time-cons…
Knowledge DistillationGraph Neural Network