Predictive Modeling through Hyper-Bayesian Optimization
Model selection is an integral problem of model based optimization techniques such as Bayesian optimization (BO). Current approaches often treat model selection as an estimation problem, to be periodically updated with observations coming from the optimization iterations. In this paper, we propose an alternative way to achieve both efficiently. Specifically, we propose a novel way of integrating model selection and BO for the single goal of reaching the function optima faster. The algorithm moves back and forth between BO in the model space and BO in the function space, where the goodness of the recommended model is captured by a score function and fed back, capturing how well the model helped convergence in the function space. The score function is derived in such a way that it neutralizes the effect of the moving nature of the BO in the function space, thus keeping the model selection problem stationary. This back and forth leads to quick convergence for both model selection and BO in the function space. In addition to improved sample efficiency, the framework outputs information about the black-box function. Convergence is proved, and experimental results show significant improvement compared to standard BO.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationModel SelectionSimilar Papers 제목 키워드 기반
The Sensitivity of Variational Bayesian Neural Network Performance to Hyperparameters
In scientific applications, predictive modeling is often of limited use without accurate uncertainty quantification (UQ) to indicate when a model may be extrapolating or when more data needs to be collected. Bayesian Neu…
Dimensionality ReductionInformed Initialization for Bayesian Optimization and Active Learning
Bayesian Optimization is a widely used method for optimizing expensive black-box functions, relying on probabilistic surrogate models such as Gaussian Processes. The quality of the surrogate model is crucial for good opt…
Gaussian ProcessesActive LearningPFNs4BO: In-Context Learning for Bayesian Optimization
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) throug…
AutoMLBayesian OptimizationHyperparameter OptimizationIn-Context LearningFLASH: Fast Bayesian Optimization for Data Analytic Pipelines
Modern data science relies on data analytic pipelines to organize interdependent computational steps. Such analytic pipelines often involve different algorithms across multiple steps, each with its own hyperparameters. T…
Bayesian OptimizationPrior Specification for Bayesian Matrix Factorization via Prior Predictive Matching
The behavior of many Bayesian models used in machine learning critically depends on the choice of prior distributions, controlled by some hyperparameters that are typically selected by Bayesian optimization or cross-vali…
Bayesian OptimizationHyperparameter OptimizationStochastic Optimization