Hyperparameter Optimization via Sequential Uniform Designs
Hyperparameter optimization (HPO) plays a central role in the automated machine learning (AutoML). It is a challenging task as the response surfaces of hyperparameters are generally unknown, hence essentially a global optimization problem. This paper reformulates HPO as a computer experiment and proposes a novel sequential uniform design (SeqUD) strategy with three-fold advantages: a) the hyperparameter space is adaptively explored with evenly spread design points, without the need of expensive meta-modeling and acquisition optimization; b) the batch-by-batch design points are sequentially generated with parallel processing support; c) a new augmented uniform design algorithm is developed for the efficient real-time generation of follow-up design points. Extensive experiments are conducted on both global optimization tasks and HPO applications. The numerical results show that the proposed SeqUD strategy outperforms benchmark HPO methods, and it can be therefore a promising and competitive alternative to existing AutoML tools.
Code (2)
Tasks
AutoMLglobal-optimizationHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Open Loop Hyperparameter Optimization and Determinantal Point Processes
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evalua…
DiversityHyperparameter OptimizationPoint ProcessesYou Only Train Once
The title of this paper is perhaps an overclaim. Of course, the process of creating and optimizing a learned model inevitably involves multiple training runs which potentially feature different architectural designs, inp…
Semantic SegmentationInformed Initialization for Bayesian Optimization and Active Learning
Bayesian Optimization is a widely used method for optimizing expensive black-box functions, relying on probabilistic surrogate models such as Gaussian Processes. The quality of the surrogate model is crucial for good opt…
Gaussian ProcessesActive LearningHyp-RL : Hyperparameter Optimization by Reinforcement Learning
Hyperparameter tuning is an omnipresent problem in machine learning as it is an integral aspect of obtaining the state-of-the-art performance for any model. Most often, hyperparameters are optimized just by training a mo…
Bayesian OptimizationHyperparameter Optimizationreinforcement-learningReinforcement Learning+1Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization
The remarkable performance of modern deep learning methods depends critically on the optimization of their hyperparameters. One major challenge is that evaluating a single hyperparameter configuration on large datasets c…
Gaussian ProcessesHyperparameter Optimization