Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
Code (1)
Tasks
Bayesian OptimizationGaussian ProcessesHyperparameter OptimizationSimilar Papers 제목 키워드 기반
Improving Multi-fidelity Optimization with a Recurring Learning Rate for Hyperparameter Tuning
Despite the evolution of Convolutional Neural Networks (CNNs), their performance is surprisingly dependent on the choice of hyperparameters. However, it remains challenging to efficiently explore large hyperparameter sea…
image-classificationImage ClassificationTransfer LearningA Diffusion-Based Framework for Configurable and Realistic Multi-Storage Trace Generation
We propose DiTTO, a novel diffusion-based framework for generating realistic, precisely configurable, and diverse multi-device storage traces. Leveraging advanced diffusion techniques, DiTTO enables the synthesis of high…
Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning
Bayesian optimization is popular for optimizing time-consuming black-box objectives. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyper…
Bayesian OptimizationEfficient Hyperparameter Optimization for LLM Reinforcement Learning
Reinforcement learning (RL) for large language models (LLMs) is highly sensitive to hyperparameter configurations, making hyperparameter optimization (HPO) essential yet computationally expensive. Existing multi-fidelity…
Hyperparameter OptimizationComputational EfficiencyReinforcement LearningPhysics-Informed CoKriging: A Gaussian-Process-Regression-Based Multifidelity Method for Data-Model Convergence
In this work, we propose a new Gaussian process regression (GPR)-based multifidelity method: physics-informed CoKriging (CoPhIK). In CoKriging-based multifidelity methods, the quantities of interest are modeled as linear…
Active LearningGaussian ProcessesGPRregression