Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets
Bayesian optimization (BO) is widely adopted in black-box optimization problems and it relies on a surrogate model to approximate the black-box response function. With the increasing number of black-box optimization tasks solved and even more to solve, the ability to learn from multiple prior tasks to jointly pre-train a surrogate model is long-awaited to further boost optimization efficiency. In this paper, we propose a simple approach to pre-train a surrogate, which is a Gaussian process (GP) with a kernel defined on deep features learned from a Transformer-based encoder, using datasets from prior tasks with possibly heterogeneous input spaces. In addition, we provide a simple yet effective mix-up initialization strategy for input tokens corresponding to unseen input variables and therefore accelerate new tasks' convergence. Experiments on both synthetic and real benchmark problems demonstrate the effectiveness of our proposed pre-training and transfer BO strategy over existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels
Bayesian Optimization (BO), guided by Gaussian process (GP) surrogates, has proven to be an invaluable technique for efficient, high-dimensional, black-box optimization, a critical problem inherent to many applications s…
Bayesian OptimizationMeta-Learningreinforcement-learningReinforcement Learning+1Bayesian Optimization with Approximate Set Kernels
We propose a practical Bayesian optimization method over sets, to minimize a black-box function that takes a set as a single input. Because set inputs are permutation-invariant, traditional Gaussian process-based Bayesia…
Bayesian OptimizationKernel-based guarantees for nonlinear parametric models in Bayesian optimization
Modern Bayesian optimization and adaptive sampling methods increasingly rely on nonlinear parametric models, yet theoretical guarantees for such models under adaptive data collection remain limited. Existing analyses lar…
Gaussian ProcessesEnd-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization
For Bayesian optimization (BO) on high-dimensional data with complex structure, neural network-based kernels for Gaussian processes (GPs) have been used to learn flexible surrogate functions by the high representation po…
Bayesian OptimizationGaussian ProcessesMeta-LearningBayesian Optimization for auto-tuning GPU kernels
Finding optimal parameter configurations for tunable GPU kernels is a non-trivial exercise for large search spaces, even when automated. This poses an optimization task on a non-convex search space, using an expensive to…
Bayesian OptimizationGPU