Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters
Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to find the best optima. Surprisingly, there is little theoretical analysis of this crucial problem in the literature. To address this, we derive a cumulative regret bound for Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expected improvement acquisition function and sub-Gaussian observation noise, provides us with guidelines on how to design hyper-parameter estimation methods. A simple simulation demonstrates the importance of following these guidelines.
Code (1)
Tasks
Bayesian OptimisationGaussian Processesparameter estimationSimilar Papers 제목 키워드 기반
Time-Varying Gaussian Process Bandits with Unknown Prior
Bayesian optimisation requires fitting a Gaussian process model, which in turn requires specifying prior on the unknown black-box function -- most of the theoretical literature assumes this prior is known. However, it is…
Bayesian OptimisationSub-linear Regret Bounds for Bayesian Optimisation in Unknown Search Spaces
Bayesian optimisation is a popular method for efficient optimisation of expensive black-box functions. Traditionally, BO assumes that the search space is known. However, in many problems, this assumption does not hold. T…
Bayesian OptimisationBatch Bayesian optimisation via density-ratio estimation with guarantees
Bayesian optimisation (BO) algorithms have shown remarkable success in applications involving expensive black-box functions. Traditionally BO has been set as a sequential decision-making process which estimates the utili…
Bayesian InferenceBayesian OptimisationDecision MakingDensity Ratio Estimation+1Mean-Field Bayesian Optimisation
We address the problem of optimising the average payoff for a large number of cooperating agents, where the payoff function is unknown and treated as a black box. While standard Bayesian Optimisation (BO) methods struggl…
Bayesian OptimisationAdjoint-aided inference of Gaussian process driven differential equations
Linear systems occur throughout engineering and the sciences, most notably as differential equations. In many cases the forcing function for the system is unknown, and interest lies in using noisy observations of the sys…
Bayesian InferenceBayesian Optimisation