Transformer Learns Optimal Variable Selection in Group-Sparse Classification
Transformers have demonstrated remarkable success across various applications. However, the success of transformers have not been understood in theory. In this work, we give a case study of how transformers can be trained to learn a classic statistical model with "group sparsity", where the input variables form multiple groups, and the label only depends on the variables from one of the groups. We theoretically demonstrate that, a one-layer transformer trained by gradient descent can correctly leverage the attention mechanism to select variables, disregarding irrelevant ones and focusing on those beneficial for classification. We also demonstrate that a well-pretrained one-layer transformer can be adapted to new downstream tasks to achieve good prediction accuracy with a limited number of samples. Our study sheds light on how transformers effectively learn structured data.
Code (0)
등록된 구현이 없습니다.
Tasks
Variable SelectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Optimal variable selection in multi-group sparse discriminant analysis
This article considers the problem of multi-group classification in the setting where the number of variables $p$ is larger than the number of observations $n$. Several methods have been proposed in the literature that a…
Variable SelectionGrouped Variable Selection for Generalized Eigenvalue Problems
Many problems require the selection of a subset of variables from a full set of optimization variables. The computational complexity of an exhaustive search over all possible subsets of variables is, however, prohibitive…
Variable SelectionVariable Selection and Task Grouping for Multi-Task Learning
We consider multi-task learning, which simultaneously learns related prediction tasks, to improve generalization performance. We factorize a coefficient matrix as the product of two matrices based on a low-rank assumptio…
Multi-Task LearningVariable SelectionFeature selection with optimal coordinate ascent (OCA)
In machine learning, Feature Selection (FS) is a major part of efficient algorithm. It fuels the algorithm and is the starting block for our prediction. In this paper, we present a new method, called Optimal Coordinate A…
feature selectionGrouped Orthogonal Matching Pursuit for Variable Selection and Prediction
We consider the problem of variable group selection for least squares regression, namely, that of selecting groups of variables for best regression performance, leveraging and adhering to a natural grouping structure wit…
feature selectionPredictionregressionVariable Selection