Error Controlled Actor-Critic
On error of value function inevitably causes an overestimation phenomenon and has a negative impact on the convergence of the algorithms. To mitigate the negative effects of the approximation error, we propose Error Controlled Actor-critic which ensures confining the approximation error in value function. We present an analysis of how the approximation error can hinder the optimization process of actor-critic methods.Then, we derive an upper boundary of the approximation error of Q function approximator and find that the error can be lowered by restricting on the KL-divergence between every two consecutive policies when training the policy. The results of experiments on a range of continuous control tasks demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and significantly outperforms other model-free RL algorithms.
Code (1)
Tasks
continuous-controlContinuous ControlSimilar Papers 제목 키워드 기반
Error Controlled Actor-Critic Method to Reinforcement Learning
In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably cause overestimation phenomenon and have a negative impact on the conve…
continuous-controlContinuous ControlOpenAI Gymreinforcement-learning+2Finite-Time Analysis of Natural Actor-Critic for POMDPs
We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of t…
Goal-Oriented Interference Coordination in 6G In-Factory Subnetworks
Subnetworks are expected to enhance wireless pervasiveness for critical applications such as wireless control of plants, however, they are interference-limited due to their extreme density. This paper proposes a goal-ori…
Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning
Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or value loss. Critic complexity is introduced as an additional diagnostic …
Reinforcement LearningCritic-Guided Decoding for Controlled Text Generation
Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as…
Language ModelingLanguage Modellingreinforcement-learningReinforcement Learning+2