Why long model-based rollouts are no reason for bad Q-value estimates

2024-07-16 · Philipp Wissmann, Daniel Hein, Steffen Udluft, Volker Tresp

This paper explores the use of model-based offline reinforcement learning with long model rollouts. While some literature criticizes this approach due to compounding errors, many practitioners have found success in real-world applications. The paper aims to demonstrate that long rollouts do not necessarily result in exponentially growing errors and can actually produce better Q-value estimates than model-free methods. These findings can potentially enhance reinforcement learning techniques.

📄 PDF Abstract BibTeX arXiv:2407.11751

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations

2023-08-07 · Nirbhay Modhe, Qiaozi Gao, Ashwin Kalyan, Dhruv Batra 외

Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions. Model-free methods penalize values at …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Double Horizon Model-Based Policy Optimization

2025-12-17 · Akihiro Kubo, Paavo Parmas, Shin Ishii arxiv

Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a learned dynamics model. However, choosing the length of the rollouts p…

Reinforcement Learning

Learning to Plan via Deep Optimistic Value Exploration

2020-06-08 · L4DC 2020 6 · Tim Seyde, Wilko Schwarting, Sertac Karaman, Daniela Rus

Deep exploration requires coordinated long-term planning. We present a model-based reinforcement learning algorithm that guides policy learning through a value function that exhibits optimism in the face of uncertainty. …

Benchmarking Model-based Reinforcement Learning Reinforcement Learning (RL)

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

2026-05-08 · Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh 외 arxiv

Reinforcement learning with verifiable rewards (RLVR) for Large Reasoning Models hinges on baseline estimation for variance reduction, but existing approaches pay a heavy price: PPO requires a policy-model scale critic, …

Reinforcement Learning

ProAct: Agentic Lookahead in Interactive Environments

2026-02-05 · Yangbin Yu, Mingyu Yang, Junyou Li, Yiming Gao 외 arxiv

Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a fr…