Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery
Multi-index models provide a popular framework to investigate the learnability of functions with low-dimensional structure and, also due to their connections with neural networks, they have been object of recent intensive study. In this paper, we focus on recovering the subspace spanned by the signals via spectral estimators -- a family of methods routinely used in practice, often as a warm-start for iterative algorithms. Our main technical contribution is a precise asymptotic characterization of the performance of spectral methods, when sample size and input dimension grow proportionally and the dimension $p$ of the space to recover is fixed. Specifically, we locate the top-$p$ eigenvalues of the spectral matrix and establish the overlaps between the corresponding eigenvectors (which give the spectral estimators) and a basis of the signal subspace. Our analysis unveils a phase transition phenomenon in which, as the sample complexity grows, eigenvalues escape from the bulk of the spectrum and, when that happens, eigenvectors recover directions of the desired subspace. The precise characterization we put forward enables the optimization of the data preprocessing, thus allowing to identify the spectral estimator that requires the minimal sample size for weak recovery.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning single index model with gradient descent: spectral initialization and precise asymptotics
Non-convex optimization plays a central role in many statistics and machine learning problems. Despite the landscape irregularities for general non-convex functions, some recent work showed that for many learning problem…
Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models
In a mixed generalized linear model, the objective is to learn multiple signals from unlabeled observations: each sample comes from exactly one signal, but it is not known which one. We consider the prototypical problem …
RetrievalSpectral and post-spectral estimators for grouped panel data models
In this paper, we develop spectral and post-spectral estimators for grouped panel data models. Both estimators are consistent in the asymptotics where the number of observations $N$ and the number of time periods $T$ sim…
Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers
The problem of learning one task with samples from another task is central to transfer learning (TL). In this paper, we examine a fundamental question: When does combining the data samples from a source task and a target…
Multi-Task Learningtext-classificationText ClassificationTransfer LearningUncertainty quantification in the Bradley-Terry-Luce model
The Bradley-Terry-Luce (BTL) model is a benchmark model for pairwise comparisons between individuals. Despite recent progress on the first-order asymptotics of several popular procedures, the understanding of uncertainty…
modelUncertainty Quantification