Automatic Model Monitoring for Data Streams
Detecting concept drift is a well known problem that affects production systems. However, two important issues that are frequently not addressed in the literature are 1) the detection of drift when the labels are not immediately available; and 2) the automatic generation of explanations to identify possible causes for the drift. For example, a fraud detection model in online payments could show a drift due to a hot sale item (with an increase in false positives) or due to a true fraud attack (with an increase in false negatives) before labels are available. In this paper we propose SAMM, an automatic model monitoring system for data streams. SAMM detects concept drift using a time and space efficient unsupervised streaming algorithm and it generates alarm reports with a summary of the events and features that are important to explain it. SAMM was evaluated in five real world fraud detection datasets, each spanning periods up to eight months and totaling more than 22 million online transactions. We evaluated SAMM using human feedback from domain experts, by sending them 100 reports generated by the system. Our results show that SAMM is able to detect anomalous events in a model life cycle that are considered useful by the domain experts. Given these results, SAMM will be rolled out in a next version of Feedzai's Fraud Detection solution.
Code (0)
등록된 구현이 없습니다.
Tasks
Fraud DetectionmodelSimilar Papers 제목 키워드 기반
Unsupervised Behavior Change Detection in Multidimensional Data Streams for Maritime Traffic Monitoring
The worldwide growth of maritime traffic and the development of the Automatic Identification System (AIS) has led to advances in monitoring systems for preventing vessel accidents and detecting illegal activities. In thi…
BIG-bench Machine LearningChange DetectionEvent DetectionSemi-supervised Change DetectionA Multi-task Deep Learning Architecture for Maritime Surveillance using AIS Data Streams
In a world of global trading, maritime safety, security and efficiency are crucial issues. We propose a multi-task deep learning framework for vessel monitoring using Automatic Identification System (AIS) data streams. W…
Anomaly DetectionDeep LearningThe SUMMA Platform Prototype
We present the first prototype of the SUMMA Platform: an integrated platform for multilingual media monitoring. The platform contains a rich suite of low-level and high-level natural language processing technologies: aut…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)General ClassificationMachine Translation+4Online Tool Condition Monitoring Based on Parsimonious Ensemble+
Accurate diagnosis of tool wear in metal turning process remains an open challenge for both scientists and industrial practitioners because of inhomogeneities in workpiece material, nonstationary machining settings to su…
Active LearningEnsemble Learningfeature selectionStream-Based Monitoring of Algorithmic Fairness
Automatic decision and prediction systems are increasingly deployed in applications where they significantly impact the livelihood of people, such as for predicting the creditworthiness of loan applicants or the recidivi…
FairnessPrediction