Hyperparameter Adaptive Search for Surrogate Optimization: A Self-Adjusting Approach
Surrogate Optimization (SO) algorithms have shown promise for optimizing expensive black-box functions. However, their performance is heavily influenced by hyperparameters related to sampling and surrogate fitting, which poses a challenge to their widespread adoption. We investigate the impact of hyperparameters on various SO algorithms and propose a Hyperparameter Adaptive Search for SO (HASSO) approach. HASSO is not a hyperparameter tuning algorithm, but a generic self-adjusting SO algorithm that dynamically tunes its own hyperparameters while concurrently optimizing the primary objective function, without requiring additional evaluations. The aim is to improve the accessibility, effectiveness, and convergence speed of SO algorithms for practitioners. Our approach identifies and modifies the most influential hyperparameters specific to each problem and SO approach, reducing the need for manual tuning without significantly increasing the computational burden. Experimental results demonstrate the effectiveness of HASSO in enhancing the performance of various SO algorithms across different global optimization test problems.
Code (0)
등록된 구현이 없습니다.
Tasks
global-optimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Scalable Hyperparameter Optimization with Products of Gaussian Process Experts
In machine learning, hyperparameter optimization is a challenging but necessary task that is usually approached in a computationally expensive manner such as grid-search. Out of this reason, surrogate based black-box opt…
Gaussian ProcessesHyperparameter OptimizationDynamic Surrogate Switching: Sample-Efficient Search for Factorization Machine Configurations in Online Recommendations
Hyperparameter optimization is the process of identifying the appropriate hyperparameter configuration of a given machine learning model with regard to a given learning task. For smaller data sets, an exhaustive search i…
Hyperparameter OptimizationMulti-level CNN for lung nodule classification with Gaussian Process assisted hyperparameter optimization
This paper investigates lung nodule classification by using deep neural networks (DNNs). Hyperparameter optimization in DNNs is a computationally expensive problem, where evaluating a hyperparameter configuration may tak…
Bayesian OptimizationGaussian ProcessesGeneral ClassificationHyperparameter Optimization+1Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparamete…
Bayesian OptimizationGaussian ProcessesHyperparameter OptimizationSMAC+1FastBO: Fast HPO and NAS with Adaptive Fidelity Identification
Hyperparameter optimization (HPO) and neural architecture search (NAS) are powerful in attaining state-of-the-art machine learning models, with Bayesian optimization (BO) standing out as a mainstream method. Extending BO…
Bayesian OptimizationHyperparameter OptimizationNeural Architecture SearchNovel Concepts