Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous Controlmodelreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients
Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a hig…
continuous-controlContinuous ControlDeep Reinforcement LearningMuJoCo+2Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods
In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overes…
continuous-controlContinuous ControlDeep Reinforcement LearningOpenAI Gym+3Policy Optimization with Model-based Explorations
Model-free reinforcement learning methods such as the Proximal Policy Optimization algorithm (PPO) have successfully applied in complex decision-making problems such as Atari games. However, these methods suffer from hig…
Atari GamesDecision MakingmodelModel-based Reinforcement Learning+3Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion
By integrating dynamics models into model-free reinforcement learning (RL) methods, model-based value expansion (MVE) algorithms have shown a significant advantage in sample efficiency as well as value estimation. Howeve…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Model-Free Reinforcement Learning with the Decision-Estimation Coefficient
We consider the problem of interactive decision making, encompassing structured bandits and reinforcement learning with general function approximation. Recently, Foster et al. (2021) introduced the Decision-Estimation Co…
Decision Makingreinforcement-learningReinforcement Learning