Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

2020-09-21 · Jun-Jie Wang, Qichao Zhang, Dongbin Zhao, Mengchen Zhao, Jianye Hao

Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection, we utilize a world model with a reconstruction module for image feature extraction, in order to acquire more precise value estimation. The raw and the reconstructed images are both used to determine the appropriate horizon for adaptive value expansion. On several benchmark visual control tasks, experimental results show that DMVE outperforms all baselines in sample efficiency and final performance, indicating that DMVE can achieve more effective and accurate value estimation than state-of-the-art model-based methods.

📄 PDF Abstract BibTeX arXiv:2009.09593

Code (0)

등록된 구현이 없습니다.

Tasks

Model-based Reinforcement LearningNovelty Detectionreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Horizon Adaptive Offline Policy Learning via Value Stitching

2026-06-19 · Kexin Zheng, Xianyuan Zhan, Xintao Yan arxiv

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation …

Reinforcement Learning Offline RL

ViVa: A Video-Generative Value Model for Robot Reinforcement Learning

2026-04-09 · Jindi Lv, Hao Li, Jie Li, Fankun Kong 외 arxiv

Vision-language-action (VLA) models have advanced robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback. Reinforcement learnin…

Reinforcement Learning Robot Manipulation

Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning

2024-02-05 · Zihan Ding, Amy Zhang, Yuandong Tian, Qinqing Zheng

We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently. As opposed to traditional one-step dynamics models, DWM offers long-horizon …

D4RL Q-Learning

A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

2026-03-30 · Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen arxiv

Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned dynamics models for policy optimization. However, the effectiveness of methods such as actor-critic is often limited by compoundi…

Reinforcement Learning Continuous Control Offline RL

Double Horizon Model-Based Policy Optimization

2025-12-17 · Akihiro Kubo, Paavo Parmas, Shin Ishii arxiv

Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a learned dynamics model. However, choosing the length of the rollouts p…

Reinforcement Learning