Generative Bayesian Hyperparameter Tuning
\noindent Hyper-parameter selection is a central practical problem in modern machine learning, governing regularization strength, model capacity, and robustness choices. Cross-validation is often computationally prohibitive at scale, while fully Bayesian hyper-parameter learning can be difficult due to the cost of posterior sampling. We develop a generative perspective on hyper-parameter tuning that combines two ideas: (i) optimization-based approximations to Bayesian posteriors via randomized, weighted objectives (weighted Bayesian bootstrap), and (ii) amortization of repeated optimization across many hyper-parameter settings by learning a transport map from hyper-parameters (including random weights) to the corresponding optimizer. This yields a ``generator look-up table'' for estimators, enabling rapid evaluation over grids or continuous ranges of hyper-parameters and supporting both predictive tuning objectives and approximate Bayesian uncertainty quantification. We connect this viewpoint to weighted $M$-estimation, envelope/auxiliary-variable representations that reduce non-quadratic losses to weighted least squares, and recent generative samplers for weighted $M$-estimators.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Fast Hyperparameter Tuning using Bayesian Optimization with Directional Derivatives
In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound …
Bayesian OptimizationLearning TheoryPAC learningSmall Data Image ClassificationWeighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
Many contemporary machine learning models require extensive tuning of hyperparameters to perform well. A variety of methods, such as Bayesian optimization, have been developed to automate and expedite this process. Howev…
Bayesian OptimizationPractical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning
Bayesian optimization is popular for optimizing time-consuming black-box objectives. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyper…
Bayesian OptimizationTowards Automatic Bayesian Optimization: A first step involving acquisition functions
Bayesian Optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression nor its gradients, they are expensive to evaluate an…
Bayesian OptimizationBIG-bench Machine LearningTowards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tun…
Bayesian OptimizationBIG-bench Machine LearningHyperparameter OptimizationNeural Architecture Search