Bias Correction in Deterministic Policy Gradient Using Robust MPC
In this paper, we discuss the deterministic policy gradient using the Actor-Critic methods based on the linear compatible advantage function approximator, where the input spaces are continuous. When the policy is restricted by hard constraints, the exploration may not be Centred or Isotropic (non-CI). As a result, the policy gradient estimation can be biased. We focus on constrained policies based on Model Predictive Control (MPC) schemes and to address the bias issue, we propose an approximate Robust MPC approach accounting for the exploration. The RMPC-based policy ensures that a Centered and Isotropic (CI) exploration is approximately feasible. A posterior projection is used to ensure its exact feasibility, we formally prove that this approach does not bias the gradient estimation.
Code (0)
등록된 구현이 없습니다.
Tasks
Model Predictive ControlSimilar Papers 제목 키워드 기반
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during…
Reinforcement LearningMitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach
Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when th…
continuous-controlContinuous ControlDeep Reinforcement LearningQ-Learning+2Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients
It is vital to accurately estimate the value function in Deep Reinforcement Learning (DRL) such that the agent could execute proper actions instead of suboptimal ones. However, existing actor-critic methods suffer more o…
continuous-controlContinuous ControlDeep Reinforcement LearningSoftmax Deep Double Deterministic Policy Gradients
A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Althou…
continuous-controlContinuous ControlDeterministic Value-Policy Gradients
Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexi…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+2