Variational Model-based Policy Optimization

2020-06-09 · Yin-Lam Chow, Brandon Cui, Moonkyung Ryu, Mohammad Ghavamzadeh

Model-based reinforcement learning (RL) algorithms allow us to combine model-generated data with those collected from interaction with the real system in order to alleviate the data efficiency problem in RL. However, designing such algorithms is often challenging because the bias in simulated data may overshadow the ease of data generation. A potential solution to this challenge is to jointly learn and improve model and policy using a universal objective function. In this paper, we leverage the connection between RL and probabilistic inference, and formulate such an objective function as a variational lower-bound of a log-likelihood. This allows us to use expectation maximization (EM) and iteratively fix a baseline policy and learn a variational distribution, consisting of a model and a policy (E-step), followed by improving the baseline policy given the learned variational distribution (M-step). We propose model-based and model-free policy iteration (actor-critic) style algorithms for the E-step and show how the variational distribution learned by them can be used to optimize the M-step in a fully model-based fashion. Our experiments on a number of continuous control tasks show that despite being more complex, our model-based (E-step) algorithm, called {\em variational model-based policy optimization} (VMBPO), is more sample-efficient and robust to hyper-parameter tuning than its model-free (E-step) counterpart. Using the same control tasks, we also compare VMBPO with several state-of-the-art model-based and model-free RL algorithms and show its sample efficiency and performance.

📄 PDF Abstract BibTeX arXiv:2006.05443

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlmodelModel-based Reinforcement LearningReinforcement Learning (RL)

Variational Proximal Policy Optimization

2026-06-06 · Ousmane Amadou Dia arxiv

Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Op…

Reinforcement Learning Mathematical Reasoning

Variational Policy Search via Trajectory Optimization

2013-12-01 · NeurIPS 2013 12 · Sergey Levine, Vladlen Koltun

In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task. While random exploration can work well in simple domains, comp…

Variational Policy Gradient Method for Reinforcement Learning with General Utilities

2020-07-04 · NeurIPS 2020 12 · Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari 외

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this pa…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Variational Monte Carlo

Constrained Variational Policy Optimization for Safe Reinforcement Learning

2022-01-28 · Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu 외

Safe reinforcement learning (RL) aims to learn policies that satisfy certain constraints before deploying them to safety-critical applications. Previous primal-dual style approaches suffer from instability issues and lac…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Markov Chain Monte Carlo Policy Optimization

2021-01-04 · Daniel Hsu

Discovering approximately optimal policies in domains is crucial to applying reinforcement learning (RL) in many real-world scenarios, which is termed as policy optimization. By viewing the policy optimization from the p…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning (RL)+1