PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs
Code (2)
Tasks
Deep LearningHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LMEMs for post-hoc analysis of HPO Benchmarking
The importance of tuning hyperparameters in Machine Learning (ML) and Deep Learning (DL) is established through empirical research and applications, evident from the increase in new hyperparameter optimization (HPO) algo…
BenchmarkingHyperparameter OptimizationAuto-nnU-Net: Towards Automated Medical Image Segmentation
Medical Image Segmentation (MIS) includes diverse tasks, from bone to organ segmentation, each with its own challenges in finding the best segmentation model. The state-of-the-art AutoML-related MIS-framework nnU-Net aut…
AutoMLComputational EfficiencyHyperparameter OptimizationImage Segmentation+5Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization
Many machine learning solutions are framed as optimization problems which rely on good hyperparameters. Algorithms for tuning these hyperparameters usually assume access to exact solutions to the underlying learning prob…
Practical Transfer Learning for Bayesian Optimization
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemb…
Bayesian OptimizationGaussian ProcessesHyperparameter OptimizationMeta-Learning+1Online hyperparameter optimization by real-time recurrent learning
Conventional hyperparameter optimization methods are computationally intensive and hard to generalize to scenarios that require dynamically adapting hyperparameters, such as life-long learning. Here, we propose an online…
Hyperparameter Optimization