Faster Riemannian Newton-type Optimization by Subsampling and Cubic Regularization
This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed and convergence behavior compared to a large set of state-of-the-art Riemannian optimization algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Riemannian optimizationVocal Bursts Type PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dual Riemannian Newton Method on Statistical Manifolds
In probabilistic modeling, parameter estimation is commonly formulated as a minimization problem on a parameter manifold. Optimization in such spaces requires geometry-aware methods that respect the underlying informatio…
Scalable Second-order Riemannian Optimization for $K$-means Clustering
Clustering is a hard discrete optimization problem. Nonconvex approaches such as low-rank semidefinite programming (SDP) have recently demonstrated promising statistical and local algorithmic guarantees for cluster recov…
A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks
We analyze the convergence of Gauss-Newton dynamics for training neural networks with smooth activation functions. In the underparameterized regime, the Gauss-Newton gradient flow induces a Riemannian gradient flow on a …
Riemannian optimizationAn Investigation of Newton-Sketch and Subsampled Newton Methods
Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems i…
Dimensionality ReductionDo Subsampled Newton Methods Work for High-Dimensional Data?
Subsampled Newton methods approximate Hessian matrices through subsampling techniques, alleviating the cost of forming Hessian matrices but using sufficient curvature information. However, previous results require $\Omeg…
Distributed OptimizationVocal Bursts Intensity Prediction