Fast Hyperparameter Tuning using Bayesian Optimization with Directional Derivatives
In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound will be higher for a small subset of data compared to the whole, and the highest accuracy for a small subset of data can be achieved with a simple model. We initially tune the hyperparameters on a small subset of training data using Bayesian optimization. While tuning the hyperparameters on the whole training data, we leverage the insights from the learning theory to seek more complex models. We realize this by using directional derivative signs strategically placed in the hyperparameter search space to seek a more complex model than the one obtained with small data. We demonstrate the performance of our method on the tasks of tuning the hyperparameters of several machine learning algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationLearning TheoryPAC learningSmall Data Image ClassificationSimilar Papers 제목 키워드 기반
From Black-Box Tuning to Guided Optimization via Hyperparameters Interaction Analysis
Hyperparameters tuning is a fundamental, yet computationally expensive, step in optimizing machine learning models. Beyond optimization, understanding the relative importance and interaction of hyperparameters is critica…
Scalable Differentially Private Bayesian Optimization
In recent years, there has been much work on scaling Bayesian Optimization to high-dimensional problems, for example hyperparameter tuning in large machine learning models. These scalable methods have been successful, fi…
Bayesian OptimizationWeighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
Many contemporary machine learning models require extensive tuning of hyperparameters to perform well. A variety of methods, such as Bayesian optimization, have been developed to automate and expedite this process. Howev…
Bayesian OptimizationFLASH: Fast Bayesian Optimization for Data Analytic Pipelines
Modern data science relies on data analytic pipelines to organize interdependent computational steps. Such analytic pipelines often involve different algorithms across multiple steps, each with its own hyperparameters. T…
Bayesian OptimizationTowards Automatic Bayesian Optimization: A first step involving acquisition functions
Bayesian Optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression nor its gradients, they are expensive to evaluate an…
Bayesian OptimizationBIG-bench Machine Learning