Some approaches used to overcome overestimation in Deep Reinforcement Learning algorithms

2020-06-25 · Rafael Stekolshchik

Some phenomena related to statistical noise which have been investigated by various authors under the framework of deep reinforcement learning (RL) algorithms are discussed. The following algorithms are examined: the deep Q-network (DQN), double DQN, deep deterministic policy gradient (DDPG), twin-delayed DDPG (TD3), and hill climbing algorithm. First, we consider overestimation, which is a harmful property resulting from noise. Then we deal with noise used for exploration, this is the useful noise. We discuss setting the noise parameter in the TD3 for typical PyBullet environments associated with articulate bodies such as HopperBulletEnv and Walker2DBulletEnv. In the appendix, in relation to the hill climbing algorithm, another example related to noise is considered - an example of adaptive noise.

📄 PDF Abstract BibTeX arXiv:2006.14167

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Target Policy Smoothing Target Policy Smoothing is a regularization strategy for the value function in reinforcement learning. Deterministic policies can overfit to narrow peaks in the value…

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Weight Decay 설명 없음

Adam 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

2021-09-24 · Baturay Saglam, Furkan Burak Mutlu, Dogan Can Cicek, Suleyman Serdar Kozat

Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a hig…

continuous-control Continuous Control Deep Reinforcement Learning MuJoCo+2

Double Q-learning

2010-12-01 · NeurIPS 2010 12 · Hado V. Hasselt

In some stochastic environments the well-known reinforcement learning algorithm Q-learning performs very poorly. This poor performance is caused by large overestimations of action values. These overestimations result fro…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Behavior Proximal Policy Optimization

2023-02-22 · Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang 외

Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to the overestimation of out-of-distribution state-action pairs. Thus, various additional aug…

D4RL Offline RL Reinforcement Learning (RL)

Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods

2021-09-22 · Baturay Saglam, Enes Duran, Dogan C. Cicek, Furkan B. Mutlu 외

In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overes…

continuous-control Continuous Control Deep Reinforcement Learning OpenAI Gym+3

Cross Learning in Deep Q-Networks

2020-09-29 · Xing Wang, Alexander Vinel

In this work, we propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods, particularly in the deep Q-networks where the overestim…

Q-Learning reinforcement-learning Reinforcement Learning (RL)