Model-based Reinforcement Learning with Multi-step Plan Value Estimation

2022-09-12 · Haoxin Lin, Yihao Sun, Jiaji Zhang, Yang Yu

A promising way to improve the sample efficiency of reinforcement learning is model-based methods, in which many explorations and evaluations can happen in the learned models to save real-world samples. However, when the learned model has a non-negligible model error, sequential steps in the model are hard to be accurately evaluated, limiting the model's utilization. This paper proposes to alleviate this issue by introducing multi-step plans to replace multi-step actions for model-based RL. We employ the multi-step plan value estimation, which evaluates the expected discounted return after executing a sequence of action plans at a given state, and updates the policy by directly computing the multi-step policy gradient via plan value estimation. The new model-based reinforcement learning algorithm MPPVE (Model-based Planning Policy Learning with Multi-step Plan Value Estimation) shows a better utilization of the learned model and achieves a better sample efficiency than state-of-the-art model-based RL approaches.

📄 PDF Abstract BibTeX arXiv:2209.05530

Code (1)

HxLyn3/MPPVE 공식 구현 pytorch

Tasks

Model-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Combating the Compounding-Error Problem with a Multi-step Model

2019-05-30 · Kavosh Asadi, Dipendra Misra, Seungchan Kim, Michel L. Littman

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a sta…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning

2020-06-23 · Lingheng Meng, Rob Gorbet, Dana Kulić

Multi-step (also called n-step) methods in reinforcement learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in ta…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

2022-01-24 · ICLR 2022 4 · Haichao Zhang, Wei Xu, Haonan Yu

Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties ar…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

How to Combine Tree-Search Methods in Reinforcement Learning

2018-09-06 · Yonathan Efroni, Gal Dalal, Bruno Scherrer, Shie Mannor

Finite-horizon lookahead policies are abundantly used in Reinforcement Learning and demonstrate impressive empirical success. Usually, the lookahead policies are implemented with specific planning methods such as Monte C…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Gamma-Nets: Generalizing Value Estimation over Timescale

2019-11-18 · Craig Sherstan, Shibhansh Dohare, James Macglashan, Johannes Günther 외

We present $\Gamma$-nets, a method for generalizing value function estimation over timescale. By using the timescale as one of the estimator's inputs we can estimate value for arbitrary timescales. As a result, the predi…

Deep Reinforcement Learning Lifelong learning Reinforcement Learning Representation Learning