TreeGrad: Transferring Tree Ensembles to Neural Networks

2019-04-25 · Chapman Siu

Gradient Boosting Decision Tree (GBDT) are popular machine learning algorithms with implementations such as LightGBM and in popular machine learning toolkits like Scikit-Learn. Many implementations can only produce trees in an offline manner and in a greedy manner. We explore ways to convert existing GBDT implementations to known neural network architectures with minimal performance loss in order to allow decision splits to be updated in an online manner and provide extensions to allow splits points to be altered as a neural architecture search problem. We provide learning bounds for our neural network.

📄 PDF Abstract BibTeX arXiv:1904.11132

Code (1)

chappers/TreeGrad 공식 구현 tf

Tasks

BIG-bench Machine Learning Neural Architecture Search

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

TreeGrad-Ranker: Feature Ranking via $O(L)$-Time Gradients for Decision Trees

2026-02-12 · Weida Li, Yaoliang Yu, Bryan Kian Hsiang Low arxiv

We revisit the use of probabilistic values, which include the well-known Shapley and Banzhaf values, to rank features for explaining the local predicted values of decision trees. The quality of feature rankings is typica…

ForestPrune: Compact Depth-Controlled Tree Ensembles

2022-05-31 · Brian Liu, Rahul Mazumder

Tree ensembles are powerful models that achieve excellent predictive performances, but can grow to unwieldy sizes. These ensembles are often post-processed (pruned) to reduce memory footprint and improve interpretability…

Flexible Modeling and Multitask Learning using Differentiable Tree Ensembles

2022-05-19 · Shibal Ibrahim, Hussein Hazimeh, Rahul Mazumder

Decision tree ensembles are widely used and competitive learning models. Despite their success, popular toolkits for learning tree ensembles have limited modeling capabilities. For instance, these toolkits support a limi…

Multi-Task Learning

MotherNets: Rapid Deep Ensemble Learning

2018-09-12 · Abdul Wasay, Brian Hentschel, Yuze Liao, Sanyuan Chen 외

Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either …

Clustering Clustering Ensemble Ensemble Learning Knowledge Distillation

On Computing Optimal Tree Ensembles

2023-06-07 · Christian Komusiewicz, Pascal Kunz, Frank Sommer, Manuel Sorge

Random forests and, more generally, (decision\nobreakdash-)tree ensembles are widely used methods for classification and regression. Recent algorithmic advances allow to compute decision trees that are optimal for variou…