Towards Uncertainties in Deep Learning that Are Accurate and Calibrated
Predictive uncertainties can be characterized by two properties---calibration and sharpness. This paper introduces algorithms that ensure the calibration of any model while maintaining sharpness. They apply in both classification and regression and guarantee the strong property of distribution calibration, while being simpler and more broadly applicable than previous methods (especially in the context of neural networks, which are often miscalibrated). Importantly, these algorithms achieve a long-standing statistical principle that forecasts should maximize sharpness subject to being fully calibrated. Using our algorithms, machine learning models can under some assumptions be calibrated without sacrificing accuracy: in a sense, calibration can be a free lunch. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningregressionSimilar Papers 제목 키워드 기반
Calibrated Model-Based Deep Reinforcement Learning
Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inacc…
Deep Reinforcement LearningmodelModel-based Reinforcement LearningMuJoCo+3Calibrated Probabilistic Forecasts for Arbitrary Sequences
Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors, which challenges the validity of forecasts. We present a forecasting framework ensuring valid uncertaint…
Decision MakingvalidCalibrated Perception Uncertainty Across Objects and Regions in Bird's-Eye-View
In driving scenarios with poor visibility or occlusions, it is important that the autonomous vehicle would take into account all the uncertainties when making driving decisions, including choice of a safe speed. The grid…
Semantic SegmentationAccurate Uncertainties for Deep Learning Using Calibrated Regression
Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. Bayesian methods provide a general framework to quantify uncertainty. However, because of model misspeci…
Deep LearningModel-based Reinforcement LearningregressionReinforcement Learning+4Online Calibrated and Conformal Prediction Improves Bayesian Optimization
Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.…
Bayesian OptimizationConformal PredictionDecision MakingHyperparameter Optimization+1