Bayesian Optimization with Conformal Prediction Sets
Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects decisions (i.e. objective function queries) with maximal expected utility with respect to the posterior distribution of a Bayesian model, which quantifies reducible, epistemic uncertainty about query outcomes. In practice, subjectively implausible outcomes can occur regularly for two reasons: 1) model misspecification and 2) covariate shift. Conformal prediction is an uncertainty quantification method with coverage guarantees even for misspecified models and a simple mechanism to correct for covariate shift. We propose conformal Bayesian optimization, which directs queries towards regions of search space where the model predictions have guaranteed validity, and investigate its behavior on a suite of black-box optimization tasks and tabular ranking tasks. In many cases we find that query coverage can be significantly improved without harming sample-efficiency.
Code (1)
Tasks
Active LearningBayesian OptimizationConformal PredictionDecision MakingDecision Making Under UncertaintyPredictionPrediction IntervalsUncertainty QuantificationSimilar Papers 제목 키워드 기반
On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning
Bayesian deep learning and conformal prediction are two methods that have been used to convey uncertainty and increase safety in machine learning systems. We focus on combining Bayesian deep learning with split conformal…
Conformal PredictionDeep Learningimage-classificationImage Classification+2Bayesian Conformal Prediction as a Decision Risk Problem
We propose Bayesian Conformal Prediction (BCP), a framework that combines Bayesian posterior predictive distributions with PAC-style conformal risk control to produce prediction sets with finite-sample coverage guarantee…
Robust Bayesian Optimization via Localized Online Conformal Prediction
Bayesian optimization (BO) is a sequential approach for optimizing black-box objective functions using zeroth-order noisy observations. In BO, Gaussian processes (GPs) are employed as probabilistic surrogate models to es…
Bayesian OptimizationConformal PredictionGaussian ProcessesPredictionEfficiency of conformalized ridge regression
Conformal prediction is a method of producing prediction sets that can be applied on top of a wide range of prediction algorithms. The method has a guaranteed coverage probability under the standard IID assumption regard…
Conformal PredictionPredictionPrediction IntervalsregressionConformalized Kernel Ridge Regression
General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confide…
Anomaly Detectionregression