ControlBurn: Feature Selection by Sparse Forests
Tree ensembles distribute feature importance evenly amongst groups of correlated features. The average feature ranking of the correlated group is suppressed, which reduces interpretability and complicates feature selection. In this paper we present ControlBurn, a feature selection algorithm that uses a weighted LASSO-based feature selection method to prune unnecessary features from tree ensembles, just as low-intensity fire reduces overgrown vegetation. Like the linear LASSO, ControlBurn assigns all the feature importance of a correlated group of features to a single feature. Moreover, the algorithm is efficient and only requires a single training iteration to run, unlike iterative wrapper-based feature selection methods. We show that ControlBurn performs substantially better than feature selection methods with comparable computational costs on datasets with correlated features.
Code (1)
Tasks
Feature Importancefeature selectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ControlBurn: Nonlinear Feature Selection with Sparse Tree Ensembles
ControlBurn is a Python package to construct feature-sparse tree ensembles that support nonlinear feature selection and interpretable machine learning. The algorithms in this package first build large tree ensembles that…
Additive modelsfeature selectionInterpretable Machine LearningEnd-to-end Feature Selection Approach for Learning Skinny Trees
We propose a new optimization-based approach for feature selection in tree ensembles, an important problem in statistics and machine learning. Popular tree ensemble toolkits e.g., Gradient Boosted Trees and Random Forest…
Ensemble LearningFeature CompressionFeature Importancefeature selection+1Markov Network Structure Learning via Ensemble-of-Forests Models
Real world systems typically feature a variety of different dependency types and topologies that complicate model selection for probabilistic graphical models. We introduce the ensemble-of-forests model, a generalization…
Model SelectionNonparametric Feature Selection by Random Forests and Deep Neural Networks
Random forests are a widely used machine learning algorithm, but their computational efficiency is undermined when applied to large-scale datasets with numerous instances and useless features. Herein, we propose a nonpar…
BIG-bench Machine LearningComputational Efficiencyfeature selectionSparse Projection Oblique Randomer Forests
Decision forests, including Random Forests and Gradient Boosting Trees, have recently demonstrated state-of-the-art performance in a variety of machine learning settings. Decision forests are typically ensembles of axis-…
Computational Efficiency