Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods

2021-09-22 · Baturay Saglam, Enes Duran, Dogan C. Cicek, Furkan B. Mutlu, Suleyman S. Kozat

In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overestimation bias, if the reinforcement signals received by the agent have a high variance, a significant underestimation bias arises. To minimize the underestimation, we introduce a parameter-free, novel deep Q-learning variant. Our Q-value update rule combines the notions behind Clipped Double Q-learning and Maxmin Q-learning by computing the critic objective through the nested combination of maximum and minimum operators to bound the approximate value estimates. We evaluate our modification on the suite of several OpenAI Gym continuous control tasks, improving the state-of-the-art in every environment tested.

📄 PDF Abstract BibTeX arXiv:2109.10736

Code (1)

baturaysaglam/TCD3 pytorch

Tasks

continuous-control Continuous ControlDeep Reinforcement LearningOpenAI GymQ-Learningreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Clipped Double Q-learning 설명 없음

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning

2024-09-28 · Haohui Chen, Zhiyong Chen, Aoxiang Liu, Wentuo Fang

To obtain better value estimation in reinforcement learning, we propose a novel algorithm based on the double actor-critic framework with temporal difference error-driven regularization, abbreviated as TDDR. TDDR employs…

continuous-control Continuous Control

Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

2021-09-24 · Baturay Saglam, Furkan Burak Mutlu, Dogan Can Cicek, Suleyman Serdar Kozat

Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a hig…

continuous-control Continuous Control Deep Reinforcement Learning MuJoCo+2

Actor-Critic Reinforcement Learning with Phased Actor

2024-04-18 · Ruofan Wu, Junmin Zhong, Jennie Si

Policy gradient methods in actor-critic reinforcement learning (RL) have become perhaps the most promising approaches to solving continuous optimal control problems. However, the trial-and-error nature of RL and the inhe…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method

2025-10-01 · Andy Wu, Chun-Cheng Lin, Rung-Tzuo Liaw, Yuehua Huang 외 arxiv

Reinforcement learning has gathered much attention in recent years due to its rapid development and rich applications, especially on control systems and robotics. When tackling real-world applications with reinforcement …

Reinforcement Learning

Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients

2021-12-21 · Jiafei Lyu, Yu Yang, Jiangpeng Yan, Xiu Li

It is vital to accurately estimate the value function in Deep Reinforcement Learning (DRL) such that the agent could execute proper actions instead of suboptimal ones. However, existing actor-critic methods suffer more o…

continuous-control Continuous Control Deep Reinforcement Learning