An Empirical Study of Uncertainty Estimation Techniques for Detecting Drift in Data Streams
In safety-critical domains such as autonomous driving and medical diagnosis, the reliability of machine learning models is crucial. One significant challenge to reliability is concept drift, which can cause model deterioration over time. Traditionally, drift detectors rely on true labels, which are often scarce and costly. This study conducts a comprehensive empirical evaluation of using uncertainty values as substitutes for error rates in detecting drifts, aiming to alleviate the reliance on labeled post-deployment data. We examine five uncertainty estimation methods in conjunction with the ADWIN detector across seven real-world datasets. Our results reveal that while the SWAG method exhibits superior calibration, the overall accuracy in detecting drifts is not notably impacted by the choice of uncertainty estimation method, with even the most basic method demonstrating competitive performance. These findings offer valuable insights into the practical applicability of uncertainty-based drift detection in real-world, safety-critical applications.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingDrift DetectionMedical DiagnosisSimilar Papers 제목 키워드 기반
Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning
We consider the problem of detecting out-of-distribution (OOD) samples in deep reinforcement learning. In a value based reinforcement learning setting, we propose to use uncertainty estimation techniques directly on the …
Bayesian InferenceDeep Reinforcement LearningOpen-Ended Question AnsweringOut-of-Distribution Detection+4Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection
Deep learning techniques show success in detecting objects in medical images, but still suffer from false-positive predictions that may hinder accurate diagnosis. The estimated uncertainty of the neural network output ha…
Lesion DetectionReference-free Hallucination Detection for Large Vision-Language Models
Large vision-language models (LVLMs) have made significant progress in recent years. While LVLMs exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, they are prone…
HallucinationQuestion AnsweringUncertainty QuantificationSources of Uncertainty in 3D Scene Reconstruction
The process of 3D scene reconstruction can be affected by numerous uncertainty sources in real-world scenes. While Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (GS) achieve high-fidelity rendering, they lack …
3D Reconstruction3D Scene ReconstructionNeRFThe Peril of Popular Deep Learning Uncertainty Estimation Methods
Uncertainty estimation (UE) techniques -- such as the Gaussian process (GP), Bayesian neural networks (BNN), Monte Carlo dropout (MCDropout) -- aim to improve the interpretability of machine learning models by assigning …
Deep Learning