Calibration by Distribution Matching: Trainable Kernel Calibration Metrics
Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.
Code (1)
Tasks
Decision MakingregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Calibration Regularized Training of Deep Neural Networks using Kernel Density Estimation
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. This property is particularly important in safety-critical applications such as medical …
Autonomous DrivingDensity EstimationMedical DiagnosisTrainable Calibration Measures for Neural Networks from Kernel Mean Embeddings
Modern neural networks have recently been found to be poorly calibrated, primarily in the direction of over-confidence. Methods like entropy penalty and temperature smoothing improve calibration by clamping confiden…
A Consistent and Differentiable Lp Canonical Calibration Error Estimator
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to o…
Simple Calibration via Geodesic Kernels
Deep discriminative approaches, such as decision forests and deep neural networks, have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical …
Out-of-Distribution DetectionregressionAll Models Are Miscalibrated, But Some Less So: Comparing Calibration with Conditional Mean Operators
When working in a high-risk setting, having well calibrated probabilistic predictive models is a crucial requirement. However, estimators for calibration error are not always able to correctly distinguish which model is …
All