Combating the Compounding-Error Problem with a Multi-step Model

2019-05-30 · Kavosh Asadi, Dipendra Misra, Seungchan Kim, Michel L. Littman

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.

📄 PDF Abstract BibTeX arXiv:1905.13320

Code (0)

등록된 구현이 없습니다.

Tasks

Model-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Directly Forecasting Belief for Reinforcement Learning with Delays

2025-05-01 · Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan, YiXuan Wang 외

Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (S…

D4RL MuJoCo reinforcement-learning Reinforcement Learning+1

Investigating Compounding Prediction Errors in Learned Dynamics Models

2022-03-17 · Nathan Lambert, Kristofer Pister, Roberto Calandra

Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and predic…

Model-based Reinforcement Learning Prediction

Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error

2025-04-02 · Anne Somalwar, Bruce D. Lee, George J. Pappas, Nikolai Matni

Compounding error, where small prediction mistakes accumulate over time, presents a major challenge in learning-based control. For example, this issue often limits the performance of model-based reinforcement learning an…

Imitation Learning Model-based Reinforcement Learning

Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors

2020-06-08 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor K. Prasanna

Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited whe…

model Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Curriculum-Based Neighborhood Sampling For Sequence Prediction

2018-09-16 · James O' Neill, Danushka Bollegala

The task of multi-step ahead prediction in language models is challenging considering the discrepancy between training and testing. At test time, a language model is required to make predictions given past predictions as…

Language Modeling Language Modelling Prediction