Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

2021-09-24 · Baturay Saglam, Furkan Burak Mutlu, Dogan Can Cicek, Suleyman Serdar Kozat

Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a high variance, deep actor-critic approaches that overcome the overestimation bias lead to a substantial underestimation bias. We first address the detrimental issues in the existing approaches that aim to overcome such underestimation error. Then, through extensive statistical analysis, we introduce a novel, parameter-free Deep Q-learning variant to reduce this underestimation bias in deterministic policy gradients. By sampling the weights of a linear combination of two approximate critics from a highly shrunk estimation bias interval, our Q-value update rule is not affected by the variance of the rewards received by the agents throughout learning. We test the performance of the introduced improvement on a set of MuJoCo and Box2D continuous control tasks and demonstrate that it considerably outperforms the existing approaches and improves the state-of-the-art by a significant margin.

📄 PDF Abstract BibTeX arXiv:2109.11788

Code (1)

baturaysaglam/swtd3 공식 구현 pytorch

Tasks

continuous-control Continuous ControlDeep Reinforcement LearningMuJoCoQ-LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Test 설명 없음

Clipped Double Q-learning 설명 없음

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Automating Control of Overestimation Bias for Reinforcement Learning

2021-10-26 · Arsenii Kuznetsov, Alexander Grishin, Artem Tsypin, Arsenii Ashukha 외

Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defined bias correction policies that are eit…

Continuous Control Q-Learning reinforcement-learning Reinforcement Learning+1

Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods

2021-09-22 · Baturay Saglam, Enes Duran, Dogan C. Cicek, Furkan B. Mutlu 외

In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overes…

continuous-control Continuous Control Deep Reinforcement Learning OpenAI Gym+3

Reward-estimation variance elimination in sequential decision processes

2018-11-15 · Sergey Pankov

Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor…

Policy Gradient Methods Reinforcement Learning

AWD3: Dynamic Reduction of the Estimation Bias

2021-11-12 · Dogan C. Cicek, Enes Duran, Baturay Saglam, Kagan Kaya 외

Value-based deep Reinforcement Learning (RL) algorithms suffer from the estimation bias primarily caused by function approximation and temporal difference (TD) learning. This problem induces faulty state-action value est…

continuous-control Continuous Control Deep Reinforcement Learning OpenAI Gym+1

Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

2026-04-23 · Ziyang Wei, Wanrong Zhu, Jingyang Lyu, Wei Biao Wu arxiv

We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require …