Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
We present a simple and powerful algorithm for parallel black box optimization called Successive Halving and Classification (SHAC). The algorithm operates in $K$ stages of parallel function evaluations and trains a cascade of binary classifiers to iteratively cull the undesirable regions of the search space. SHAC is easy to implement, requires no tuning of its own configuration parameters, is invariant to the scale of the objective function and can be built using any choice of binary classifier. We adopt tree-based classifiers within SHAC and achieve competitive performance against several strong baselines for optimizing synthetic functions, hyperparameters and architectures.
Code (1)
Tasks
ClassificationGeneral ClassificationSimilar Papers 제목 키워드 기반
Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems
On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-base…
Hyperparameter OptimizationSchedulingSuccessive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes
Successive Halving is a popular algorithm for hyperparameter optimization which allocates exponentially more resources to promising candidates. However, the algorithm typically relies on intermediate performance values t…
Hyperparameter OptimizationGaussian ProcessesMulti-objective Asynchronous Successive Halving
Hyperparameter optimization (HPO) is increasingly used to automatically tune the predictive performance (e.g., accuracy) of machine learning models. However, in a plethora of real-world applications, accuracy is only one…
FairnessHyperparameter OptimizationLanguage ModelingLanguage Modelling+2Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity
We propose a new algorithm for hyperparameter selection in machine learning algorithms. The algorithm is a novel modification of Harmonica, a spectral hyperparameter selection approach using sparse recovery methods. In p…
BIG-bench Machine LearningHyperparameter OptimizationA resource-efficient method for repeated HPO and NAS problems
In this work we consider the problem of repeated hyperparameter and neural architecture search (HNAS). We propose an extension of Successive Halving that is able to leverage information gained in previous HNAS problems w…
Neural Architecture SearchTransfer Learning