Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
Random forests remain among the most popular off-the-shelf supervised machine learning tools with a well-established track record of predictive accuracy in both regression and classification settings. Despite their empirical success as well as a bevy of recent work investigating their statistical properties, a full and satisfying explanation for their success has yet to be put forth. Here we aim to take a step forward in this direction by demonstrating that the additional randomness injected into individual trees serves as a form of implicit regularization, making random forests an ideal model in low signal-to-noise ratio (SNR) settings. Specifically, from a model-complexity perspective, we show that the mtry parameter in random forests serves much the same purpose as the shrinkage penalty in explicitly regularized regression procedures like lasso and ridge regression. To highlight this point, we design a randomized linear-model-based forward selection procedure intended as an analogue to tree-based random forests and demonstrate its surprisingly strong empirical performance. Numerous demonstrations on both real and synthetic data are provided.
Code (1)
Tasks
regressionSimilar Papers 제목 키워드 기반
Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization
In this work we show how to use the Operational Space Control framework (OSC) under joint and cartesian constraints for reinforcement learning in cartesian space. Our method is therefore able to learn fast and with adjus…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Degrees of Freedom in Deep Neural Networks
In this paper, we explore degrees of freedom in deep sigmoidal neural networks. We show that the degrees of freedom in these models is related to the expected optimism, which is the expected difference between test error…
General ClassificationMulti-class ClassificationWhen Does More Regularization Imply Fewer Degrees of Freedom? Sufficient Conditions and Counter Examples from Lasso and Ridge Regression
Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., be…
regressionRandomization of Short-Rate Models, Analytic Pricing and Flexibility in Controlling Implied Volatilities
We focus on extending existing short-rate models, enabling control of the generated implied volatility while preserving analyticity. We achieve this goal by applying the Randomized Affine Diffusion (RAnD) method to the c…
Domain Randomization for Robust, Affordable and Effective Closed-loop Control of Soft Robots
Soft robots are gaining popularity thanks to their intrinsic safety to contacts and adaptability. However, the potentially infinite number of Degrees of Freedom makes their modeling a daunting task, and in many cases onl…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)