Weighted Random Search for Hyperparameter Optimization
We introduce an improved version of Random Search (RS), used here for hyperparameter optimization of machine learning algorithms. Unlike the standard RS, which generates for each trial new values for all hyperparameters, we generate new values for each hyperparameter with a probability of change. The intuition behind our approach is that a value that already triggered a good result is a good candidate for the next step, and should be tested in new combinations of hyperparameter values. Within the same computational budget, our method yields better results than the standard RS. Our theoretical results prove this statement. We test our method on a variation of one of the most commonly used objective function for this class of problems (the Grievank function) and for the hyperparameter optimization of a deep learning CNN architecture. Our results can be generalized to any optimization problem defined on a discrete domain.
Code (1)
Tasks
Hyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Weighted Random Search for CNN Hyperparameter Optimization
Nearly all model algorithms used in machine learning use two different sets of parameters: the training parameters and the meta-parameters (hyperparameters). While the training parameters are learned during the training …
Hyperparameter OptimizationA Population-based Hybrid Approach to Hyperparameter Optimization for Neural Networks
In recent years, large amounts of data have been generated, and computer power has kept growing. This scenario has led to a resurgence in the interest in artificial neural networks. One of the main challenges in training…
Bayesian OptimizationHyperparameter OptimizationOpen Loop Hyperparameter Optimization and Determinantal Point Processes
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evalua…
DiversityHyperparameter OptimizationPoint ProcessesLearning to Warm-Start Bayesian Hyperparameter Optimization
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a lo…
Bayesian Optimizationglobal-optimizationHyperparameter OptimizationMetric LearningQuantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
Reinforcement learning algorithms can show strong variation in performance between training runs with different random seeds. In this paper we explore how this affects hyperparameter optimization when the goal is to find…
Bayesian OptimizationDeep Reinforcement LearningHyperparameter Optimizationreinforcement-learning+2