Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges
Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for supervised machine learning, can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization. This work is accompanied by an appendix that contains information on specific software packages in R and Python, as well as information and recommended hyperparameter search spaces for specific learning algorithms. We also provide notebooks that demonstrate concepts from this work as supplementary files.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationBIG-bench Machine LearningEvolutionary AlgorithmsHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hyperparameters in Reinforcement Learning and How To Tune Them
In order to improve reproducibility, deep reinforcement learning (RL) has been adopting better scientific practices such as standardized evaluation metrics and reporting. However, the process of hyperparameter optimizati…
AutoMLDeep Reinforcement LearningHyperparameter Optimizationreinforcement-learning+2Non-stochastic Best Arm Identification and Hyperparameter Optimization
Motivated by the task of hyperparameter optimization, we introduce the non-stochastic best-arm identification problem. Within the multi-armed bandit literature, the cumulative regret objective enjoys algorithms and analy…
Hyperparameter OptimizationHow to tune the RBF SVM hyperparameters?: An empirical evaluation of 18 search algorithms
SVM with an RBF kernel is usually one of the best classification algorithms for most data sets, but it is important to tune the two hyperparameters $C$ and $\gamma$ to the data itself. In general, the selection of the hy…
Bayesian OptimizationTwo-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset
Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic -- primarily relying on manual or grid searches. This is partly because adopting advance…
Hyperparameter OptimizationRethinking the Foundations for Continual Reinforcement Learning
Algorithms and approaches for continual reinforcement learning have gained increasing attention. Much of this early progress rests on the foundations and standard practices of traditional reinforcement learning, without …
Continual Learningreinforcement-learningReinforcement Learning