DEEP-BO for Hyperparameter Optimization of Deep Networks
The performance of deep neural networks (DNN) is very sensitive to the particular choice of hyper-parameters. To make it worse, the shape of the learning curve can be significantly affected when a technique like batchnorm is used. As a result, hyperparameter optimization of deep networks can be much more challenging than traditional machine learning models. In this work, we start from well known Bayesian Optimization solutions and provide enhancement strategies specifically designed for hyperparameter optimization of deep networks. The resulting algorithm is named as DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO easily outperforms or shows comparable performance with some of the well-known solutions including GP-Hedge, Hyperband, BOHB, Median Stopping Rule, and Learning Curve Extrapolation. The code used is made publicly available at https://github.com/snu-adsl/DEEP-BO.
Code (1)
Tasks
Bayesian OptimizationHyperparameter OptimizationSimilar Papers 제목 키워드 기반
Stochastic Hyperparameter Optimization through Hypernetworks
Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of wei…
BIG-bench Machine LearningHyperparameter OptimizationStochastic OptimizationOptimizing Large-Scale Hyperparameters via Automated Learning Algorithm
Modern machine learning algorithms usually involve tuning multiple (from one to thousands) hyperparameters which play a pivotal role in terms of model generalizability. Black-box optimization and gradient-based algorithm…
Hyperparameter OptimizationDP-HyPO: An Adaptive Private Hyperparameter Optimization Framework
Hyperparameter optimization, also known as hyperparameter tuning, is a widely recognized technique for improving model performance. Regrettably, when training private ML models, many practitioners often overlook the priv…
Hyperparameter OptimizationPrivacy PreservingBayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax
Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consumi…
Bayesian OptimizationGaussian ProcessesGraph ClassificationHyperparameter OptimizationDP-HyPO: An Adaptive Private Framework for Hyperparameter Optimization
Hyperparameter optimization, also known as hyperparameter tuning, is a widely recognized technique for improving model performance. Regrettably, when training private ML models, many practitioners often overlook the priv…