Feature Bagging Provides Stability

2026-07-29 · Yuheng Ma, Qiang Sun arxiv

We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.

📄 PDF Abstract BibTeX arXiv:2607.26964

Code (0)

등록된 구현이 없습니다.

Bagging Provides Assumption-free Stability

2023-01-30 · Jake A. Soloff, Rina Foygel Barber, Rebecca Willett

Bagging is an important technique for stabilizing machine learning models. In this paper, we derive a finite-sample guarantee on the stability of bagging for any model. Our result places no assumptions on the distributio…

Improving reproducibility by controlling random seed stability in machine learning based estimation via bagging

2026-04-20 · Nicholas Williams, Alejandro Schuler arxiv

Predictions from machine learning algorithms can vary across random seeds, inducing instability in downstream debiased machine learning estimators. We formalize random seed stability via a concentration condition and pro…

Stacking and stability

2019-01-26 · Nino Arsov, Martin Pavlovski, Ljupco Kocarev

Stacking is a general approach for combining multiple models toward greater predictive accuracy. It has found various application across different domains, ensuing from its meta-learning nature. Our understanding, nevert…

Meta-Learning

KNN Ensembles for Tweedie Regression: The Power of Multiscale Neighborhoods

2017-07-29 · Colleen M. Farrelly

Very few K-nearest-neighbor (KNN) ensembles exist, despite the efficacy of this approach in regression, classification, and outlier detection. Those that do exist focus on bagging features, rather than varying k or baggi…

Outlier Detection regression Topological Data Analysis

Active Transfer Bagging: A New Approach for Accelerated Active Learning Acquisition of Data by Combined Transfer Learning and Bagging Based Models

2026-02-02 · Vivienne Pelletier, Daniel J. Rivera, Obinna Nwokonkwo, Steven A. Wilson 외 arxiv

Modern machine learning has achieved remarkable success on many problems, but this success often depends on the existence of large, labeled datasets. While active learning can dramatically reduce labeling cost when annot…

Transfer Learning Active Learning