Data-Driven Threshold Machine: Scan Statistics, Change-Point Detection, and Extreme Bandits
We present a novel distribution-free approach, the data-driven threshold machine (DTM), for a fundamental problem at the core of many learning tasks: choose a threshold for a given pre-specified level that bounds the tail probability of the maximum of a (possibly dependent but stationary) random sequence. We do not assume data distribution, but rather relying on the asymptotic distribution of extremal values, and reduce the problem to estimate three parameters of the extreme value distributions and the extremal index. We specially take care of data dependence via estimating extremal index since in many settings, such as scan statistics, change-point detection, and extreme bandits, where dependence in the sequence of statistics can be significant. Key features of our DTM also include robustness and the computational efficiency, and it only requires one sample path to form a reliable estimate of the threshold, in contrast to the Monte Carlo sampling approach which requires drawing a large number of sample paths. We demonstrate the good performance of DTM via numerical examples in various dependent settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Change Point DetectionComputational EfficiencySimilar Papers 제목 키워드 기반
SecureScan: An AI-Driven Multi-Layer Framework for Malware and Phishing Detection Using Logistic Regression and Threat Intelligence Integration
The growing sophistication of modern malware and phishing campaigns has diminished the effectiveness of traditional signature-based intrusion detection systems. This work presents SecureScan, an AI-driven, triple-layer d…
Intrusion DetectionScan $B$-Statistic for Kernel Change-Point Detection
Detecting the emergence of an abrupt change-point is a classic problem in statistics and machine learning. Kernel-based nonparametric statistics have been used for this task which enjoy fewer assumptions on the distribut…
Change Point DetectionUnsupervised Parameter-free Outlier Detection using HDBSCAN* Outlier Profiles
In machine learning and data mining, outliers are data points that significantly differ from the dataset and often introduce irrelevant information that can induce bias in its statistics and models. Therefore, unsupervis…
Density EstimationOutlier DetectionMaat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective Malware Detection
The malware analysis and detection research community relies on the online platform VirusTotal to label Android apps based on the scan results of around 60 antiviral scanners. Unfortunately, there are no standards on how…
Malware AnalysisMalware DetectionDetecting, identifying, and localizing radiological material in urban environments using scan statistics
A method is proposed, based on scan statistics, to detect, identify, and localize illicit radiological material using mobile sensors in an urban environment. Our method handles varying levels of background radiation that…