Optimization without Retraction on the Random Generalized Stiefel Manifold
Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP). Solving these problems is typically done by iterative methods that require a fully formed $B$. We propose a cheap stochastic iterative method that solves the optimization problem while having access only to random estimates of $B$. Our method does not enforce the constraint in every iteration; instead, it produces iterations that converge to critical points on the generalized Stiefel manifold defined in expectation. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian optimization counterparts that require the full matrix $B$. Experiments demonstrate its effectiveness in various machine learning applications involving generalized orthogonality constraints, including CCA, ICA, and the GEVP.
Code (1)
Tasks
Riemannian optimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or em…
Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning. This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expens…
Riemannian optimizationA second-order method landing on the Stiefel manifold via Newton$\unicode{x2013}$Schulz iteration
Retraction-free approaches offer attractive low-cost alternatives to Riemannian methods on the Stiefel manifold, but they are often first-order, which may limit the efficiency under high-accuracy requirements. To this en…
Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints
Many classical and modern machine learning algorithms require solving optimization tasks under orthogonality constraints. Solving these tasks with feasible methods requires a gradient descent update followed by a retract…
Riemannian optimization with a preconditioning scheme on the generalized Stiefel manifold
Optimization problems on the generalized Stiefel manifold (and products of it) are prevalent across science and engineering. For example, in computational science they arise in symmetric (generalized) eigenvalue problems…
Dimensionality ReductionRiemannian optimization