Revisiting Model-based Value Expansion
Model-based value expansion methods promise to improve the quality of value function targets and, thereby, the effectiveness of value function learning. However, to date, these methods are being outperformed by Dyna-style algorithms with conceptually simpler 1-step value function targets. This shows that in practice, the theoretical justification of value expansion does not seem to hold. We provide a thorough empirical study to shed light on the causes of failure of value expansion methods in practice which is believed to be the compounding model error. By leveraging GPU based physics simulators, we are able to efficiently use the true dynamics for analysis inside the model-based reinforcement learning loop. Performing extensive comparisons between true and learned dynamics sheds light into this black box. This paper provides a better understanding of the actual problems in value expansion. We provide future directions of research by empirically testing the maximum theoretical performance of current approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUmodelModel-based Reinforcement LearningSimilar Papers 제목 키워드 기반
Revisiting Loss Modelling for Unstructured Pruning
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intract…
Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning
Model-based reinforcement learning is one approach to increase sample efficiency. However, the accuracy of the dynamics model and the resulting compounding error over modelled trajectories are commonly regarded as key li…
continuous-controlContinuous ControlModel-based Reinforcement Learningreinforcement-learning+1DeeperImpact: Optimizing Sparse Learned Index Structures
A lot of recent work has focused on sparse learned indexes that use deep neural architectures to significantly improve retrieval quality while keeping the efficiency benefits of the inverted index. While such sparse lear…
Language ModelingLanguage ModellingRetrievalDynamic Horizon Value Estimation for Model-based Reinforcement Learning
Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potenti…
Model-based Reinforcement LearningNovelty Detectionreinforcement-learningReinforcement Learning+1Parametric analysis of COVID-19 expansion in European countries in the period of February to June 2020
The data on number of registered cases of COVID-19 disease in twenty European countries is analyzed by the least-squares fitting procedure with generic analytic functions. Three regimes of the expansion of the disease ar…