AUGMENTED POLICY GRADIENT METHODS FOR EFFICIENT REINFORCEMENT LEARNING

2019-09-25 · Kai Lagemann, Gregor Roering, Christoph Henke, Rene Vossen, Frank Hees

We propose a new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory rollouts performed on a learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use a special type of uncertainty quantification by a stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model is addressed by next state predictions based on an uncertainty aware ensemble of dynamics models. The influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and a model-based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Uncertainty Quantification

Solving Deep Reinforcement Learning Tasks with Evolution Strategies and Linear Policy Networks

2024-02-10 · Annie Wong, Jacob de Nobel, Thomas Bäck, Aske Plaat 외

Although deep reinforcement learning methods can learn effective policies for challenging problems such as Atari games and robotics tasks, algorithms are complex, and training times are often long. This study investigate…

Atari Games Deep Reinforcement Learning Q-Learning reinforcement-learning+1

Augmented Bayesian Policy Search

2024-07-05 · Mahdi Kallel, Debabrota Basu, Riad Akrour, Carlo D'Eramo

Deterministic policies are often preferred over stochastic ones when implemented on physical systems. They can prevent erratic and harmful behaviors while being easier to implement and interpret. However, in practice, ex…

Bayesian Optimization LEMMA Policy Gradient Methods reinforcement-learning+1

Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning

2020-05-18 · Donghoon Lee

Entropy augmented to reward is known to soften the greedy argmax policy to softmax policy. Entropy augmentation is reformulated and leads to a motivation to introduce an additional entropy term to the objective function …

Q-Learning

Divergence-Augmented Policy Optimization

2025-01-25 · NeurIPS 2019 12 · Qing Wang, Yingru Li, Jiechao Xiong, Tong Zhang

In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, l…

Atari Games Deep Reinforcement Learning Policy Gradient Methods reinforcement-learning+1

Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning

2019-11-28 · Ashutosh Kumar Tiwari, Sandeep Varma Nadimpalli

Model-based reinforcement learning strategies are believed to exhibit more significant sample complexity than model-free strategies to control dynamical systems,such as quadcopters.This belief that Model-based strategies…

continuous-control Continuous Control Model-based Reinforcement Learning reinforcement-learning+2