Scalable Hyperparameter Optimization with Products of Gaussian Process Experts
In machine learning, hyperparameter optimization is a challenging but necessary task that is usually approached in a computationally expensive manner such as grid-search. Out of this reason, surrogate based black-box optimization techniques such as sequential model-based optimization have been proposed which allow for a faster hyperparameter optimization. Recent research proposes to also integrate hyperparameter performances on past data sets to allow for a faster and more efficient hyperparameter optimization. In this paper, we use products of Gaussian process experts as surrogate models for hyperparameter optimization. Naturally, Gaussian processes are a decent choice as they offer good prediction accuracy as well as estimations about their uncertainty. Additionally, their hyperparameters can be tuned very effectively. However, in the light of large meta data sets, learning a single Gaussian process is not feasible as it involves inversion of a large kernel matrix. This directly limits their usefulness for hyperparameter optimization if large scale hyperparameter performances on past data sets are given. By using products of Gaussian process experts the scalability issues can be circumvented, however, this usually comes with the price of having less predictive accuracy. In our experiments, we show empirically that products of experts nevertheless perform very well compared to a variety of published surrogate models. Thus, we propose a surrogate model that performs as well as the current state of the art, is scalable to large scale meta knowledge, does not include hyperparameters itself and finally is even very easy to parallelize.
Code (1)
Tasks
Gaussian ProcessesHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Scalable Gaussian Process Hyperparameter Optimization via Coverage Regularization
Gaussian processes (GPs) are Bayesian non-parametric models popular in a variety of applications due to their accuracy and native uncertainty quantification (UQ). Tuning GP hyperparameters is critical to ensure the valid…
Conformal PredictionGaussian ProcessesHyperparameter OptimizationPrediction+1Scalable Gaussian Processes with Grid-Structured Eigenfunctions (GP-GRIEF)
We introduce a kernel approximation strategy that enables computation of the Gaussian process log marginal likelihood and all hyperparameter derivatives in O(p) time. Our GRIEF kernel consists of p eigenfunctions fo…
Bayesian InferenceGaussian ProcessesScalable Gaussian Processes with Grid-Structured Eigenfunctions (GP-GRIEF)
We introduce a kernel approximation strategy that enables computation of the Gaussian process log marginal likelihood and all hyperparameter derivatives in $\mathcal{O}(p)$ time. Our GRIEF kernel consists of $p$ eigenfun…
Bayesian InferenceGaussian ProcessesPreconditioning for Scalable Gaussian Process Hyperparameter Optimization
Gaussian process hyperparameter optimization requires linear solves with, and log-determinants of, large kernel matrices. Iterative numerical techniques are becoming popular to scale to larger datasets, relying on the co…
Gaussian ProcessesHyperparameter OptimizationBias-Free Scalable Gaussian Processes via Randomized Truncations
Scalable Gaussian Process methods are computationally attractive, yet introduce modeling biases that require rigorous study. This paper analyzes two common techniques: early truncated conjugate gradients (CG) and random …
Gaussian Processes