Investigating Compounding Prediction Errors in Learned Dynamics Models
Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.
Code (0)
등록된 구현이 없습니다.
Tasks
Model-based Reinforcement LearningPredictionSimilar Papers 제목 키워드 기반
Learning Long-Horizon Predictions for Quadrotor Dynamics
Accurate modeling of system dynamics is crucial for achieving high-performance planning and control of robotic systems. Although existing data-driven approaches represent a promising approach for modeling dynamics, their…
PredictionInvestigating Lagrangian Neural Networks for Infinite Horizon Planning in Quadrupedal Locomotion
Lagrangian Neural Networks (LNNs) present a principled and interpretable framework for learning the system dynamics by utilizing inductive biases. While traditional dynamics models struggle with compounding errors over l…
Diminishing Return of Value Expansion Methods
Model-based reinforcement learning aims to increase sample efficiency, but the accuracy of dynamics models and the resulting compounding errors are often seen as key limitations. This paper empirically investigates poten…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningMaximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited whe…
modelModel-based Reinforcement Learningreinforcement-learningReinforcement Learning+1A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning
In model-based reinforcement learning, most algorithms rely on simulating trajectories from one-step models of the dynamics learned on data. A critical challenge of this approach is the compounding of one-step prediction…
Future predictionModel-based Reinforcement Learningreinforcement-learningReinforcement Learning