No penalty no tears: Least squares in high-dimensional linear models
Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving least squares fitting and hard thresholding. The algorithms are methodologically simple to understand intuitively, computationally easy to implement efficiently, and theoretically appealing for choosing models consistently. Numerical exercises comparing our methods with penalization-based approaches in simulations and data analyses illustrate the great potential of the proposed algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Global Minima by Penalized Full-dimensional Scaling
The full-dimensional (metric, Euclidean, least squares) multidimensional scaling stress loss function is combined with a quadratic external penalty function term. The trajectory of minimizers of stress for increasing val…
Prediction bounds for higher order total variation regularized least squares
We establish adaptive results for trend filtering: least squares estimation with a penalty on the total variation of $(k-1)^{\rm th}$ order differences. Our approach is based on combining a general oracle inequality for …
Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined $n\ll p$ si…
A novel nonconvex, smooth-at-origin penalty for statistical learning
Nonconvex penalties are utilized for regularization in high-dimensional statistical learning algorithms primarily because they yield unbiased or nearly unbiased estimators for the parameters in the model. Nonconvex penal…
Variable SelectionOn the Effectiveness of Least Squares Generative Adversarial Networks
Unsupervised learning with generative adversarial networks (GANs) has proven to be hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we f…