Orthogonal Least Squares Based Fast Feature Selection for Linear Classification
An Orthogonal Least Squares (OLS) based feature selection method is proposed for both binomial and multinomial classification. The novel Squared Orthogonal Correlation Coefficient (SOCC) is defined based on Error Reduction Ratio (ERR) in OLS and used as the feature ranking criterion. The equivalence between the canonical correlation coefficient, Fisher's criterion, and the sum of the SOCCs is revealed, which unveils the statistical implication of ERR in OLS for the first time. It is also shown that the OLS based feature selection method has speed advantages when applied for greedy search. The proposed method is comprehensively compared with the mutual information based feature selection methods and the embedded methods using both synthetic and real world datasets. The results show that the proposed method is always in the top 5 among the 12 candidate methods. Besides, the proposed method can be directly applied to continuous features without discretisation, which is another significant advantage over mutual information based methods.
Code (1)
Tasks
Classificationfeature selectionGeneral ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Grouped Orthogonal Matching Pursuit for Variable Selection and Prediction
We consider the problem of variable group selection for least squares regression, namely, that of selecting groups of variables for best regression performance, leveraging and adhering to a natural grouping structure wit…
feature selectionPredictionregressionVariable SelectionSparse Linear Regression via Generalized Orthogonal Least-Squares
Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an $l_0$-constrained least-squares problem. The Orthogonal Least-Squares (O…
regressionRecovery of Sparse Signals Using Multiple Orthogonal Least Squares
We study the problem of recovering sparse signals from compressed linear measurements. This problem, often referred to as sparse recovery or sparse reconstruction, has generated a great deal of interest in recent years. …
Computational EfficiencyOptimal Randomized First-Order Methods for Least-Squares Problems
We provide an exact analysis of a class of randomized algorithms for solving overdetermined least-squares problems. We consider first-order methods, where the gradients are pre-conditioned by an approximation of the Hess…
Sampling Requirements and Accelerated Schemes for Sparse Linear Regression with Orthogonal Least-Squares
We study the problem of inferring a sparse vector from random linear combinations of its components. We propose the Accelerated Orthogonal Least-Squares (AOLS) algorithm that improves performance of the well-known Orthog…
Clusteringregression