An Improved System for Sentence-level Novelty Detection in Textual Streams
Novelty detection in news events has long been a difficult problem. A number of models performed well on specific data streams but certain issues are far from being solved, particularly in large data streams from the WWW where unpredictability of new terms requires adaptation in the vector space model. We present a novel event detection system based on the Incremental Term Frequency-Inverse Document Frequency (TF-IDF) weighting incorporated with Locality Sensitive Hashing (LSH). Our system could efficiently and effectively adapt to the changes within the data streams of any new terms with continual updates to the vector space model. Regarding miss probability, our proposed novelty detection framework outperforms a recognised baseline system by approximately 16% when evaluating a benchmark dataset from Google News.
Code (0)
등록된 구현이 없습니다.
Tasks
Event DetectionNovelty DetectionSentenceSimilar Papers 제목 키워드 기반
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities
Much of the existing work on text novelty detection has been studied at the topic level, i.e., identifying whether the topic of a document or a sentence is novel or not. Little work has been done at the fine-grained sema…
Novelty DetectionSentenceTAP-DLND 1.0 : A Corpus for Document Level Novelty Detection
Detecting novelty of an entire document is an Artificial Intelligence (AI) frontier problem that has widespread NLP applications, such as extractive document summarization, tracking development of news events, predicting…
ArticlesBenchmarkingDocument SummarizationExtractive Document Summarization+3Compose Like Humans: Jointly Improving the Coherence and Novelty for Modern Chinese Poetry Generation
Chinese poetry is an important part of worldwide culture, and classical and modern sub-branches are quite different. The former is a unique genre and has strict constraints, while the latter is very flexible in length, o…
Cultural Vocal Bursts Intensity PredictionDiversityRetrievalSentenceCND-IDS: Continual Novelty Detection for Intrusion Detection Systems
Intrusion detection systems (IDS) play a crucial role in IoT and network security by monitoring system data and alerting to suspicious activities. Machine learning (ML) has emerged as a promising solution for IDS, offeri…
Continual LearningIntrusion DetectionNovelty DetectionTime-dependent Hierarchical Dirichlet Model for Timeline Generation
Timeline Generation aims at summarizing news from different epochs and telling readers how an event evolves. It is a new challenge that combines salience ranking with novelty detection. For long-term public events, the m…
Novelty DetectionSentence