Model-based Asynchronous Hyperparameter and Neural Architecture Search
We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization. At the heart of our method is a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations. We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classification and language modelling, and report substantial speed-ups over current state-of-the-art methods. Our new methods, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication.
Code (3)
Tasks
AutoMLBayesian OptimizationHyperparameter Optimizationimage-classificationmodelNeural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Parallel Hyperparameter Optimization Of Spiking Neural Network
Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very s…
Bayesian OptimizationGPUHyperparameter OptimizationNeural Architecture SearchAsynchronous ε-Greedy Bayesian Optimisation
Batch Bayesian optimisation (BO) is a successful technique for the optimisation of expensive black-box functions. Asynchronous BO can reduce wallclock time by starting a new evaluation as soon as another finishes, thus m…
Bayesian OptimisationThompson SamplingMulti-objective Asynchronous Successive Halving
Hyperparameter optimization (HPO) is increasingly used to automatically tune the predictive performance (e.g., accuracy) of machine learning models. However, in a plethora of real-world applications, accuracy is only one…
FairnessHyperparameter OptimizationLanguage ModelingLanguage Modelling+2Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization
Bayesian optimization (BO) is a promising approach for hyperparameter optimization of deep neural networks (DNNs), where each model training can take minutes to hours. In BO, a computationally cheap surrogate model is em…
Bayesian OptimizationComputational EfficiencyHyperparameter OptimizationNeuromorphic Architecture Optimization for Task-Specific Dynamic Learning
The ability to learn and adapt in real time is a central feature of biological systems. Neuromorphic architectures demonstrating such versatility can greatly enhance our ability to efficiently process information at the …
Meta-Learning