Natural Gradient Deep Q-learning

2018-03-20 · Ethan Knight, Osher Lerner

We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outperforms DQN without target networks, and performs no worse than DQN with target networks, suggesting that NGDQN stabilizes training and can help reduce the need for additional hyperparameter tuning. We also find that NGDQN is less sensitive to hyperparameter optimization relative to DQN. Together these results suggest that natural-gradient techniques can improve value-function optimization in deep reinforcement learning.

📄 PDF Abstract BibTeX arXiv:1803.07482

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningHyperparameter OptimizationQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

True Asymptotic Natural Gradient Optimization

2017-12-22 · Yann Ollivier

We introduce a simple algorithm, True Asymptotic Natural Gradient Optimization (TANGO), that converges to a true natural gradient descent in the limit of small learning rates, without explicit Fisher matrix estimation. …

Natural Policy Gradient Methods with Parameter-based Exploration for Control Tasks

2010-12-01 · NeurIPS 2010 12 · Atsushi Miyamae, Yuichi Nagata, Isao Ono, Shigenobu Kobayashi

In this paper, we propose an efficient algorithm for estimating the natural policy gradient with parameter-based exploration; this algorithm samples directly in the parameter space. Unlike previous methods based on natur…

Policy Gradient Methods

Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

2018-03-24 · Hugh Salimbeni, Stefanos Eleftheriadis, James Hensman

The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic…

Variational Inference

Invariance Properties of the Natural Gradient in Overparametrised Systems

2022-06-30 · Jesse van Oostrum, Johannes Müller, Nihat Ay

The natural gradient field is a vector field that lives on a model equipped with a distinguished Riemannian metric, e.g. the Fisher-Rao metric, and represents the direction of steepest ascent of an objective function on …

A Geometric Understanding of Natural Gradient

2022-02-13 · Qinxun Bai, Steven Rosenberg, Wei Xu

While natural gradients have been widely studied from both theoretical and empirical perspectives, we argue that some fundamental theoretical issues regarding the existence of gradients in infinite dimensional function s…