ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY
We study the problem of using low cost to search for hyperparameter configurations in a large search space with heterogeneous evaluation cost and model quality. We propose a blended search strategy to combine the strengths of global and local search, and prioritize them on the fly with the goal of minimizing the total cost spent in finding good configurations. Our approach demonstrates robust performance for tuning both tree-based models and deep neural networks on a large AutoML benchmark, as well as superior performance in model quality, time, and resource consumption for a production NLP model fine-tuning task.
Code (0)
등록된 구현이 없습니다.
Tasks
AutoMLHyperparameter OptimizationSimilar Papers 제목 키워드 기반
Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity
We propose a new algorithm for hyperparameter selection in machine learning algorithms. The algorithm is a novel modification of Harmonica, a spectral hyperparameter selection approach using sparse recovery methods. In p…
BIG-bench Machine LearningHyperparameter OptimizationCMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which…
Bayesian OptimizationHyperparameter OptimizationDeep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
While deep neural networks (DNNs) have been increasingly applied to choice analysis showing high predictive power, it is unclear to what extent researchers can interpret economic information from DNNs. This paper demonst…
Discrete Choice ModelsHyperparameter Search in Machine Learning
We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective. Machine learning methods attempt to build models that capture some element…
BIG-bench Machine LearningHot Swapping for Online Adaptation of Optimization Hyperparameters
We describe a general framework for online adaptation of optimization hyperparameters by `hot swapping' their values during learning. We investigate this approach in the context of adaptive learning rate selection using …