Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors

2020-06-08 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor K. Prasanna

Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited when it is used to predict long horizon trajectories due to compounding errors. In this work, we propose a Dyna-style model-based reinforcement learning algorithm, which we called Maximum Entropy Model Rollouts (MEMR). To eliminate the compounding errors, we only use our model to generate single-step rollouts. Furthermore, we propose to generate \emph{diverse} model rollouts by non-uniform sampling of the environment states such that the entropy of the model rollouts is maximized. We mathematically derived the maximum entropy sampling criteria for one data case under Gaussian prior. To accomplish this criteria, we propose to utilize a prioritized experience replay. Our preliminary experiments in challenging locomotion benchmarks show that our approach achieves the same sample efficiency of the best model-based algorithms, matches the asymptotic performance of the best model-free algorithms, and significantly reduces the computation requirements of other model-based methods.

📄 PDF Abstract BibTeX arXiv:2006.04802

Code (0)

등록된 구현이 없습니다.

Tasks

modelModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Maximum Entropy Exploration Without the Rollouts

2026-03-12 · Jacob Adamczyk, Adam Kamoski, Rahul V. Kulkarni arxiv

Efficient exploration remains a central challenge in reinforcement learning, serving as a useful pretraining objective for data collection, particularly when an external reward function is unavailable. A principled formu…

Reinforcement Learning

Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning

2025-10-05 · Ziyan Wang, Zheng Wang, Xingwei Qu, Qi Cheng 외 arxiv

Reinforcement learning (RL) has become central to enhancing reasoning in large language models (LLMs). Yet on-policy algorithms such as Group Relative Policy Optimization (GRPO) often suffer in early training: noisy grad…

Reinforcement Learning

ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning

2026-02-02 · Chu Zhao, Enneng Yang, Yuting Liu, Jianzhe Zhao 외 arxiv

Test-time reinforcement learning generates multiple candidate answers via repeated rollouts and performs online updates using pseudo-labels constructed by majority voting. To reduce overhead and improve exploration, prio…

Reinforcement Learning Visual Reasoning

Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias

2026-05-08 · Rahaf Abu Hara, Vaibbhav Murarri, Claudio Zito arxiv

Existing LLM-based policy optimizers see only scalar rewards: that a policy scored 0.45, but not whether the agent got stuck in a loop, fell into a hole on the third step, or performed well on 19 out of 20 rollouts and f…

Generalized Maximum Entropy Reinforcement Learning via Reward Shaping

2021-09-29 · Feng Tao, Yongcan Cao

Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy reg…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)