Breaking the Deadly Triad with a Target Network

2021-01-21 · Shangtong Zhang, Hengshuai Yao, Shimon Whiteson

The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously. In this paper, we investigate the target network as a tool for breaking the deadly triad, providing theoretical support for the conventional wisdom that a target network stabilizes training. We first propose and analyze a novel target network update rule which augments the commonly used Polyak-averaging style update with two projections. We then apply the target network and ridge regularization in several divergent algorithms and show their convergence to regularized TD fixed points. Those algorithms are off-policy with linear function approximation and bootstrapping, spanning both policy evaluation and control, as well as both discounted and average-reward settings. In particular, we provide the first convergent linear $Q$-learning algorithms under nonrestrictive and changing behavior policies without bi-level optimization.

📄 PDF Abstract BibTeX arXiv:2101.08862

Code (1)

ShangtongZhang/DeepRL 공식 구현 pytorch

Tasks

Q-Learning

Deep Reinforcement Learning and the Deadly Triad

2018-12-06 · Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel 외

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. W…

Deep Reinforcement Learning Learning Theory reinforcement-learning Reinforcement Learning+1

Towards Characterizing Divergence in Deep Q-Learning

2019-03-21 · Joshua Achiam, Ethan Knight, Pieter Abbeel

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function…

continuous-control Continuous Control MuJoCo OpenAI Gym+2

Revisiting a Design Choice in Gradient Temporal Difference Learning

2023-08-02 · Xiaochi Qian, Shangtong Zhang

Off-policy learning enables a reinforcement learning (RL) agent to reason counterfactually about policies that are not executed and is one of the most important ideas in RL. It, however, can lead to instability when comb…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Average-Reward Off-Policy Policy Evaluation with Function Approximation

2021-01-08 · Shangtong Zhang, Yi Wan, Richard S. Sutton, Shimon Whiteson

We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is…

Target Network and Truncation Overcome The Deadly Triad in $Q$-Learning

2022-03-05 · Zaiwei Chen, John Paul Clarke, Siva Theja Maguluri

$Q$-learning with function approximation is one of the most empirically successful while theoretically mysterious reinforcement learning (RL) algorithms, and was identified in Sutton (1999) as one of the most important t…

Q-Learning reinforcement-learning Reinforcement Learning (RL)