Accelerating SGD for Distributed Deep-Learning Using Approximated Hessian Matrix
We introduce a novel method to compute a rank $m$ approximation of the inverse of the Hessian matrix in the distributed regime. By leveraging the differences in gradients and parameters of multiple Workers, we are able to efficiently implement a distributed approximation of the Newton-Raphson method. We also present preliminary results which underline advantages and challenges of second-order methods for large stochastic optimization problems. In particular, our work suggests that novel strategies for combining gradients provide further information on the loss surface.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSecond-order methodsStochastic OptimizationSimilar Papers 제목 키워드 기반
Accelerating Sinkhorn Algorithm with Sparse Newton Iterations
Computing the optimal transport distance between statistical distributions is a fundamental task in machine learning. One remarkable recent advancement is entropic regularization and the Sinkhorn algorithm, which utilize…
FAGH: Accelerating Federated Learning with Approximated Global Hessian
In federated learning (FL), the significant communication overhead due to the slow convergence speed of training the global model poses a great challenge. Specifically, a large number of communication rounds are required…
Federated LearningKronecker-factored Quasi-Newton Methods for Deep Learning
Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training par…
Deep LearningSecond-order methodsNewton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution…
Vocal Bursts Type PredictionDistributed estimation of the inverse Hessian by determinantal averaging
In distributed optimization and distributed numerical linear algebra, we often encounter an inversion bias: if we want to compute a quantity that depends on the inverse of a sum of distributed matrices, then the sum of t…
Distributed OptimizationUncertainty Quantification