A Stochastic Quasi-Newton Method with Nesterov's Accelerated Gradient
Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving large scale non-convex optimization problems in neural networks. The performance of the proposed algorithm is evaluated in Tensorflow on benchmark classification and regression problems. The results show improved performance compared to the classical second order oBFGS and oLBFGS methods and popular first order stochastic methods such as SGD and Adam. The performance with different momentum rates and batch sizes have also been illustrated.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method
Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quas…
regressionA modified limited memory Nesterov's accelerated quasi-Newton
The Nesterov's accelerated quasi-Newton (L)NAQ method has shown to accelerate the conventional (L)BFGS quasi-Newton method using the Nesterov's accelerated gradient in several neural network (NN) applications. However, t…
An Adaptive Stochastic Nesterov Accelerated Quasi Newton Method for Training RNNs
A common problem in training neural networks is the vanishing and/or exploding gradient problem which is more prominently seen in training of Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed f…
Implementation of a modified Nesterov's Accelerated quasi-Newton Method on Tensorflow
Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence spe…
Second-order methodsA Nesterov's Accelerated quasi-Newton method for Global Routing using Deep Reinforcement Learning
Deep Q-learning method is one of the most popularly used deep reinforcement learning algorithms which uses deep neural networks to approximate the estimation of the action-value function. Training of the deep Q-network (…
Deep Reinforcement LearningQ-Learningreinforcement-learningReinforcement Learning+1