Momentum in Reinforcement Learning

2019-10-21 · Nino Vieillard, Bruno Scherrer, Olivier Pietquin, Matthieu Geist

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive $q$-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates this momentum idea. Our analysis shows that this allows MoVI to average errors over successive iterations. We show that the proposed approach can be readily extended to deep learning. Specifically, we propose a simple improvement on DQN based on MoVI, and experiment it on Atari games.

📄 PDF Abstract BibTeX arXiv:1910.09322

Code (0)

등록된 구현이 없습니다.

Tasks

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Correcting Momentum in Temporal Difference Learning

2021-06-07 · Emmanuel Bengio, Joelle Pineau, Doina Precup

A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, …

Deep Reinforcement Learning Reinforcement Learning (RL)

Momentum-Based Policy Gradient Methods

2020-07-13 · ICML 2020 1 · Feihu Huang, Shangqian Gao, Jian Pei, Heng Huang

In the paper, we propose a class of efficient momentum-based policy gradient methods for the model-free reinforcement learning, which use adaptive learning rates and do not require any large batches. Specifically, we pro…

Policy Gradient Methods

Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning

2025-07-05 · Ho Jae Lee, Se Hwan Jeon, Sangbae Kim arxiv

Humans naturally swing their arms during locomotion to regulate whole-body dynamics, reduce angular momentum, and help maintain balance. Inspired by this principle, we present a limb-level multi-agent reinforcement learn…

Multi-agent Reinforcement Learning

MDPGT: Momentum-based Decentralized Policy Gradient Tracking

2021-12-06 · Zhanhong Jiang, Xian Yeow Lee, Sin Yong Tan, Kai Liang Tan 외

We propose a novel policy gradient method for multi-agent reinforcement learning, which leverages two different variance-reduction techniques and does not require large batches over iterations. Specifically, we propose a…

Multi-agent Reinforcement Learning Policy Gradient Methods reinforcement-learning Reinforcement Learning+2

Measuring Learning Progress via Gradient-Momentum Coupling

2026-05-07 · Samuel Blad, Martin Längkvist, Amy Loutfi arxiv

Measuring learning progress is essential for curiosity-driven exploration in reinforcement learning, but widely used signals such as prediction error often fail to distinguish meaningful, learnable patterns from random n…

Reinforcement Learning