Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
Robot control using reinforcement learning has become popular, but its learning process generally terminates halfway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentionally implicit underestimation or overestimation occurs, depending on the reward design in the normal states. When the episode is terminated due to task failure, the failure may be highly valued with the unintentional overestimation, and the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in practical use of TD learning to review the exception handling at termination. This paper therefore proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. In addition, the degree of underestimation is adjusted according to the degree of stationarity at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulations and real robot experiments showed that the proposed method can stably obtain the optimal policies for various tasks and reward designs. https://youtu.be/AxXr8uFOe7M
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
The Mean Square Error (MSE) is commonly utilized to estimate the solution of the optimal value function in the vast majority of offline reinforcement learning (RL) models and has achieved outstanding performance. However…
D4RLOffline RLReinforcement Learning (RL)Predictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration
Stability under model predictive control (MPC) schemes is frequently ensured by terminal ingredients. Employing a (control) Lyapunov function as the terminal cost constitutes a common choice. Learning-based methods may b…
Model Predictive ControlStability of Finite Horizon Optimisation based Control without Terminal Weight
This paper presents a stability analysis tool for model predictive control (MPC) where control action is generated by optimising a cost function over a finite horizon. Stability analysis of MPC with a limited horizon but…
Model Predictive ControlCooptimizing Safety and Performance Using Safety Value-Constrained Model Predictive Control
Autonomous systems are increasingly deployed in real-world environments, where they must achieve high performance while maintaining safety under state and input constraints. Although Model Predictive Control (MPC) provid…
Stability and performance in transient average constrained economic MPC without terminal constraints
In this paper, we investigate system theoretic properties of transient average constrained economic model predictive control (MPC) without terminal constraints. We show that the optimal open-loop solution passes by the o…
Model Predictive Control