Additive Tree-Structured Conditional Parameter Spaces in Bayesian Optimization: A Novel Covariance Function and a Fast Implementation
Bayesian optimization (BO) is a sample-efficient global optimization algorithm for black-box functions which are expensive to evaluate. Existing literature on model based optimization in conditional parameter spaces are usually built on trees. In this work, we generalize the additive assumption to tree-structured functions and propose an additive tree-structured covariance function, showing improved sample-efficiency, wider applicability and greater flexibility. Furthermore, by incorporating the structure information of parameter spaces and the additive assumption in the BO loop, we develop a parallel algorithm to optimize the acquisition function and this optimization can be performed in a low dimensional space. We demonstrate our method on an optimization benchmark function, on a neural network compression problem, on pruning pre-trained VGG16 and ResNet50 models as well as on searching activation functions of ResNet20. Experimental results show our approach significantly outperforms the current state of the art for conditional parameter optimization including SMAC, TPE and Jenatton et al. (2017).
Code (1)
Tasks
Bayesian Optimizationglobal-optimizationNeural Network CompressionSMACSMAC+Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Additive Tree-Structured Covariance Function for Conditional Parameter Spaces in Bayesian Optimization
Bayesian optimization (BO) is a sample-efficient global optimization algorithm for black-box functions which are expensive to evaluate. Existing literature on model based optimization in conditional parameter spaces are …
Bayesian Optimizationglobal-optimizationModel CompressionSMAC+1Generalized Bayesian Additive Regression Trees Models: Beyond Conditional Conjugacy
Bayesian additive regression trees have seen increased interest in recent years due to their ability to combine machine learning techniques with principled uncertainty quantification. The Bayesian backfitting algorithm u…
regressionSurvival AnalysisUncertainty QuantificationTree-Structured Boosting: Connections Between Gradient Boosted Stumps and Full Decision Trees
Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We sho…
Additive modelsGeneral ClassificationBayesian Modeling of TVP-VARs Using Regression Trees
In light of widespread evidence of parameter instability in macroeconomic models, many time-varying parameter (TVP) models have been proposed. This paper proposes a nonparametric TVP-VAR model using Bayesian additive reg…
regressionEfficient Sample-optimal Learning of Gaussian Tree Models via Sample-optimal Testing of Gaussian Mutual Information
Learning high-dimensional distributions is a significant challenge in machine learning and statistics. Classical research has mostly concentrated on asymptotic analysis of such data under suitable assumptions. While exis…