Selective Sequential Model Selection
Many model selection algorithms produce a path of fits specifying a sequence of increasingly complex models. Given such a sequence and the data used to produce them, we consider the problem of choosing the least complex model that is not falsified by the data. Extending the selected-model tests of Fithian et al. (2014), we construct p-values for each step in the path which account for the adaptive selection of the model path using the data. In the case of linear regression, we propose two specific tests, the max-t test for forward stepwise regression (generalizing a proposal of Buja and Brown (2014)), and the next-entry test for the lasso. These tests improve on the power of the saturated-model test of Tibshirani et al. (2014), sometimes dramatically. In addition, our framework extends beyond linear regression to a much more general class of parametric and nonparametric model selection problems. To select a model, we can feed our single-step p-values as inputs into sequential stopping rules such as those proposed by G'Sell et al. (2013) and Li and Barber (2015), achieving control of the familywise error rate or false discovery rate (FDR) as desired. The FDR-controlling rules require the null p-values to be independent of each other and of the non-null p-values, a condition not satisfied by the saturated-model p-values of Tibshirani et al. (2014). We derive intuitive and general sufficient conditions for independence, and show that our proposed constructions yield independent p-values.
Code (0)
등록된 구현이 없습니다.
Tasks
modelModel SelectionregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Online Selective Conformal Prediction: Errors and Solutions
In online selective conformal inference, data arrives sequentially, and prediction intervals are constructed only when an online selection rule is met. Since online selections may break the exchangeability between the se…
Conformal PredictionPredictionPrediction IntervalsvalidDetecting and Identifying Selection Structure in Sequential Data
We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. Since this selection process often distorts statistical analysis, previous work …
Inductive BiasOnline Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach
Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any s…
Drug DiscoveryAn Analysis of the 'Blind Variation and Selective Retention' Theory of Creativity
Picasso's Guernica sketches continue to provide a fruitful testing ground for examining and assessing the Blind Variation Selective Retention (BVSR) theory of creativity. Nonmonotonicity--e.g. as indicated by a lack of s…
On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
Selective state-space models (SSMs) are an emerging alternative to the Transformer, offering the unique advantage of parallel training and sequential inference. Although these models have shown promising performance on a…
State Space Models