Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals
Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to a specific operating point, making evaluation and comparison across different studies difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a simple reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper describes the corresponding algorithm, provides a theoretical analysis, and demonstrates its utility in multiple scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox.
Code (1)
Tasks
PredictionPrediction IntervalsregressionUncertainty QuantificationSimilar Papers 제목 키워드 기반
Assessment of Prediction Intervals Using Uncertainty Characteristics Curves
Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to an ad-h…
PredictionPrediction IntervalsUncertainty QuantificationTowards Uncertainty Quantification in Generative Model Learning
While generative models have become increasingly prevalent across various domains, fundamental concerns regarding their reliability persist. A crucial yet understudied aspect of these models is the uncertainty quantifica…
Conformal Prediction on Quantifying Uncertainty of Dynamic Systems
Numerous studies have focused on learning and understanding the dynamics of physical systems from video data, such as spatial intelligence. Artificial intelligence requires quantitative assessments of the uncertainty of …
Conformal PredictionOperator learningPredictionTime SeriesHow Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment
The rapid rise of large language models (LLMs) is reshaping the landscape of automatic assessment in education. While these systems demonstrate substantial advantages in adaptability to diverse question types and flexibi…
Conditional Prediction ROC Bands for Graph Classification
Graph classification in medical imaging and drug discovery requires accuracy and robust uncertainty quantification. To address this need, we introduce Conditional Prediction ROC (CP-ROC) bands, offering uncertainty quant…
ClassificationDrug DiscoveryGraph ClassificationPrediction+1