Input Warping for Bayesian Optimization of Non-stationary Functions
Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions which can be queried efficiently, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate the effects of spatially-varying length scale. We develop a methodology for automatically learning a wide family of bijective transformations or warpings of the input space using the Beta cumulative distribution function. We further extend the warping framework to multi-task Bayesian optimization so that multiple tasks can be warped into a jointly stationary space. On a set of challenging benchmark optimization tasks, we observe that the inclusion of warping greatly improves on the state-of-the-art, producing better results faster and more reliably.
Code (1)
Tasks
Bayesian OptimizationGaussian Processesglobal-optimizationSimilar Papers 제목 키워드 기반
Sparse Spectrum Warped Input Measures for Nonstationary Kernel Learning
We establish a general form of explicit, input-dependent, measure-valued warpings for learning nonstationary kernels. While stationary kernels are ubiquitous and simple to use, they struggle to adapt to functions that va…
FormGaussian ProcessesInductive BiasBayesian Optimization with Informative Covariance
Bayesian optimization is a methodology for global optimization of unknown and expensive objectives. It combines a surrogate Bayesian regression model with an acquisition function to decide where to evaluate the objective…
Bayesian OptimizationGaussian Processesglobal-optimizationregressionModernizing HEBO: a robust Bayesian optimization baseline for practical heteroskedastic and non-stationary problems
Bayesian optimization is increasingly used to guide data-efficient experimentation in chemistry, materials science, and related laboratory settings, but its practical performance depends strongly on how well surrogate-mo…
Hyperparameter OptimizationBayesian Optimization using Deep Gaussian Processes
Bayesian Optimization using Gaussian Processes is a popular approach to deal with the optimization of expensive black-box functions. However, because of the a priori on the stationarity of the covariance matrix of classi…
Bayesian OptimizationGaussian Processesglobal-optimizationDeep Compositional Spatial Models
Spatial processes with nonstationary and anisotropic covariance structure are often used when modelling, analysing and predicting complex environmental phenomena. Such processes may often be expressed as ones that have s…
Gaussian ProcessesUncertainty Quantification