Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models
Bayesian Optimization is a popular tool for tuning algorithms in automatic machine learning (AutoML) systems. Current state-of-the-art methods leverage Random Forests or Gaussian processes to build a surrogate model that predicts algorithm performance given a certain set of hyperparameter settings. In this paper, we propose a new surrogate model based on gradient boosting, where we use quantile regression to provide optimistic estimates of the performance of an unobserved hyperparameter setting, and combine this with a distance metric between unobserved and observed hyperparameter settings to help regulate exploration. We demonstrate empirically that the new method is able to outperform some state-of-the art techniques across a reasonable sized set of classification problems.
Code (0)
등록된 구현이 없습니다.
Tasks
AutoMLBayesian OptimizationGaussian ProcessesHyperparameter Optimizationquantile regressionregressionSimilar Papers 제목 키워드 기반
Hyperparameter Tuning Cookbook: A guide for scikit-learn, PyTorch, river, and spotPython
This document provides a comprehensive guide to hyperparameter tuning using spotPython for scikit-learn, PyTorch, and river. The first part introduces spotPython's surrogate model-based optimization process, while the se…
The Role of Hyperparameters in Predictive Multiplicity
This paper investigates the critical role of hyperparameters in predictive multiplicity, where different machine learning models trained on the same dataset yield divergent predictions for identical inputs. These inconsi…
FairnessHyperparameter OptimizationPredictionBenchmarking state-of-the-art gradient boosting algorithms for classification
This work explores the use of gradient boosting in the context of classification. Four popular implementations, including original GBM algorithm and selected state-of-the-art gradient boosting frameworks (i.e. XGBoost, L…
Bayesian OptimizationBenchmarkingDiversityHyperparameter OptimizationA Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting
It is already reported in the literature that the performance of a machine learning algorithm is greatly impacted by performing proper Hyper-Parameter optimization. One of the ways to perform Hyper-Parameter optimization…
Bayesian OptimizationAutomatic Gradient Boosting
Automatic machine learning performs predictive modeling with high performing machine learning tools without human interference. This is achieved by making machine learning applications parameter-free, i.e. only a dataset…
AutoMLBIG-bench Machine LearningHyperparameter OptimizationModel Selection