Efficient Regularized Piecewise-Linear Regression Trees
We present a detailed analysis of the class of regression decision tree algorithms which employ a regulized piecewise-linear node-splitting criterion and have regularized linear models at the leaves. From a theoretic standpoint, based on Rademacher complexity framework, we present new high-probability upper bounds for the generalization error for the proposed classes of regularized regression decision tree algorithms, including LASSO-type, and $\ell_{2}$ regularization for linear models at the leaves. Theoretical result are further extended by considering a general type of variable selection procedure. Furthermore, in our work we demonstrate that the class of piecewise-linear regression trees is not only numerically stable but can be made tractable via an algorithmic implementation, presented herein, as well as with the help of modern GPU technology. Empirically, we present results on multiple datasets which highlight the strengths and potential pitfalls, of the proposed tree algorithms compared to baselines which grow trees based on piecewise constant models.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUregressionVariable SelectionSimilar Papers 제목 키워드 기반
Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression
For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred ov…
regressionBayesian Additive Regression Trees with Model Trees
Bayesian Additive Regression Trees (BART) is a tree-based machine learning method that has been successfully applied to regression and classification problems. BART assumes regularisation priors on a set of trees that wo…
modelregressionCLARITree: Cholesky and Lookahead Accelerations for Regression with Interpretable Piecewise Linear Trees
Regression trees are among the most interpretable yet expressive model classes in machine learning. Historically, greedy induction has been the dominant approach for constructing well-performing regression trees. While o…
Computational EfficiencyGradient Boosting With Piece-Wise Linear Regression Trees
Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning algorithm widely used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been d…
Ensemble LearningregressionExtensions of Morse-Smale Regression with Application to Actuarial Science
The problem of subgroups is ubiquitous in scientific research (ex. disease heterogeneity, spatial distributions in ecology...), and piecewise regression is one way to deal with this phenomenon. Morse-Smale regression off…
BIG-bench Machine Learningregression