Online Calibrated and Conformal Prediction Improves Bayesian Optimization

2021-12-08 · Shachi Deshpande, Charles Marx, Volodymyr Kuleshov

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration -- i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

📄 PDF Abstract BibTeX arXiv:2112.04620

Code (0)

등록된 구현이 없습니다.

Tasks

Bayesian OptimizationConformal PredictionDecision Making Hyperparameter Optimization Prediction

Robust Bayesian Optimization via Localized Online Conformal Prediction

2024-11-26 · Dongwon Kim, Matteo Zecchin, Sangwoo Park, Joonhyuk Kang 외

Bayesian optimization (BO) is a sequential approach for optimizing black-box objective functions using zeroth-order noisy observations. In BO, Gaussian processes (GPs) are employed as probabilistic surrogate models to es…

Bayesian Optimization Conformal Prediction Gaussian Processes Prediction

Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction

2022-10-10 · Kfir M. Cohen, Sangwoo Park, Osvaldo Simeone, Shlomo Shamai

AI tools can be useful to address model deficits in the design of communication systems. However, conventional learning-based AI algorithms yield poorly calibrated decisions, unabling to quantify their outputs uncertaint…

Conformal Prediction Prediction

Self-Calibrating Conformal Prediction

2024-02-11 · Lars van der Laan, Ahmed M. Alaa

In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and in…

Binary Classification Conformal Prediction Decision Making Management+3

AI YOU Town: Make Friends and Money with Your Digital Twin

2026-07-12 · Yan Lin, Yuyang Dai, Jiahui Geng, Yuxia Wang arxiv

Existing approaches to infer user traits and generate responses consistent with a persona rely on static prompting. They lack calibrated uncertainty, ignore sequential evidence, and drift during long interactions. We pre…

SPARC: Single-Pass Scaling for Motion Forecasting with Conformal Bayesian Last Layers

2026-08-21 · Sakif Hossain, Julian Teusch, Jörg P. Müller arxiv

Human motion forecasters are increasingly accurate and fast, but reliable deployment requires uncertainty estimates that are structured, calibrated, and efficient. Bayesian and ensemble-based uncertainty estimates often …

Motion Forecasting