A Meta-Level Learning Algorithm for Sequential Hyper-Parameter Space Reduction in AutoML
AutoML platforms have numerous options for the algorithms to try for each step of the analysis, i.e., different possible algorithms for imputation, transformations, feature selection, and modelling. Finding the optimal combination of algorithms and hyper-parameter values is computationally expensive, as the number of combinations to explore leads to an exponential explosion of the space. In this paper, we present the Sequential Hyper-parameter Space Reduction (SHSR) algorithm that reduces the space for an AutoML tool with negligible drop in its predictive performance. SHSR is a meta-level learning algorithm that analyzes past runs of an AutoML tool on several datasets and learns which hyper-parameter values to filter out from consideration on a new dataset to analyze. SHSR is evaluated on 284 classification and 375 regression problems, showing an approximate 30% reduction in execution time with a performance drop of less than 0.1%.
Code (1)
Tasks
AutoMLfeature selectionImputationSimilar Papers 제목 키워드 기반
Online Nonconvex Bilevel Optimization with Bregman Divergences
Bilevel optimization methods are increasingly relevant within machine learning, especially for tasks such as hyperparameter optimization and meta-learning. Compared to the offline setting, online bilevel optimization (OB…
Bilevel OptimizationHyperparameter OptimizationMeta-LearningHyperparameter Optimization via Sequential Uniform Designs
Hyperparameter optimization (HPO) plays a central role in the automated machine learning (AutoML). It is a challenging task as the response surfaces of hyperparameters are generally unknown, hence essentially a global op…
AutoMLglobal-optimizationHyperparameter OptimizationValue Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems
Gradient-based optimization methods for hyperparameter tuning guarantee theoretical convergence to stationary solutions when for fixed upper-level variable values, the lower level of the bilevel program is strongly conve…
Look-ahead Meta Learning for Continual Learning
The continual learning problem involves training models with limited capacity to perform well on a set of an unknown number of sequentially arriving tasks. While meta-learning shows great potential for reducing interfer…
Continual LearningMeta-LearningBiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization
Evolutionary reinforcement learning (ERL) algorithms recently raise attention in tackling complex reinforcement learning (RL) problems due to high parallelism, while they are prone to insufficient exploration or model co…
Bilevel OptimizationDiversityMuJoCoreinforcement-learning+2