Soft Gradient Boosting Machine
Gradient Boosting Machine has proven to be one successful function approximator and has been widely used in a variety of areas. However, since the training procedure of each base learner has to take the sequential order, it is infeasible to parallelize the training process among base learners for speed-up. In addition, under online or incremental learning settings, GBMs achieved sub-optimal performance due to the fact that the previously trained base learners can not adapt with the environment once trained. In this work, we propose the soft Gradient Boosting Machine (sGBM) by wiring multiple differentiable base learners together, by injecting both local and global objectives inspired from gradient boosting, all base learners can then be jointly optimized with linear speed-up. When using differentiable soft decision trees as base learner, such device can be regarded as an alternative version of the (hard) gradient boosting decision trees with extra benefits. Experimental results showed that, sGBM enjoys much higher time efficiency with better accuracy, given the same base learner in both on-line and off-line settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Incremental LearningSimilar Papers 제목 키워드 기반
Infinitesimal gradient boosting
We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learni…
Soft Gradient Boosting with Learnable Feature Transforms for Sequential Regression
We propose a soft gradient boosting framework for sequential regression that embeds a learnable linear feature transform within the boosting procedure. At each boosting iteration, we train a soft decision tree and learn …
Gradient boosting machine with partially randomized decision trees
The gradient boosting machine is a powerful ensemble-based machine learning method for solving regression problems. However, one of the difficulties of its using is a possible discontinuity of the regression function, wh…
BIG-bench Machine LearningregressionEfficient Fraud Detection Using Deep Boosting Decision Trees
Fraud detection is to identify, monitor, and prevent potentially fraudulent activities from complex data. The recent development and success in AI, especially machine learning, provides a new data-driven way to deal with…
Fraud DetectionRepresentation LearningElectrical load forecasting using hybrid of extreme gradient boosting and light gradient boosting machine
Ensemble learning methods have been used to improve performance accuracy through bias-variance trade-off techniques. However, there is still room to improve. This paper proposes an ensemble model to forecast the electric…
Electrical EngineeringEnsemble LearningHybrid Machine LearningLoad Forecasting