Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks
A scaled conjugate gradient method that accelerates existing adaptive methods utilizing stochastic gradients is proposed for solving nonconvex optimization problems with deep neural networks. It is shown theoretically that, whether with constant or diminishing learning rates, the proposed method can obtain a stationary point of the problem. Additionally, its rate of convergence with diminishing learning rates is verified to be superior to that of the conjugate gradient method. The proposed method is shown to minimize training loss functions faster than the existing adaptive methods in practical applications of image and text classification. Furthermore, in the training of generative adversarial networks, one version of the proposed method achieved the lowest Frechet inception distance score among those of the adaptive methods.
Code (0)
등록된 구현이 없습니다.
Tasks
text-classificationText ClassificationSimilar Papers 제목 키워드 기반
Stochastic Conjugate Gradient Algorithm with Variance Reduction
Conjugate gradient (CG) methods are a class of important methods for solving linear equations and nonlinear optimization problems. In this paper, we propose a new stochastic CG algorithm with variance reduction and we pr…
Computational EfficiencyScaled Gradients on Grassmann Manifolds for Matrix Completion
This paper describes gradient methods based on a scaled metric on the Grassmann manifold for low-rank matrix completion. The proposed methods significantly improve canonical gradient methods especially on ill-conditioned…
Low-Rank Matrix CompletionMatrix CompletionSAGRAD: A Program for Neural Network Training with Simulated Annealing and the Conjugate Gradient Method
SAGRAD (Simulated Annealing GRADient), a Fortran 77 program for computing neural networks for classification using batch learning, is discussed. Neural network training in SAGRAD is based on a combination of simulated an…
A Newton-CG based barrier method for finding a second-order stationary point of nonconvex conic optimization with complexity guarantees
In this paper we consider finding an approximate second-order stationary point (SOSP) of nonconvex conic optimization that minimizes a twice differentiable function over the intersection of an affine subspace and a conve…
Second-order methodsAdaptive Methods for Nonconvex Optimization
Adaptive gradient methods that rely on scaling gradients down by the square root of exponential moving averages of past squared gradients, such RMSProp, Adam, Adadelta have found wide application in optimizing the noncon…
Stochastic Optimization