Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.
Code (1)
Tasks
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning
We consider the problem of detecting out-of-distribution (OOD) samples in deep reinforcement learning. In a value based reinforcement learning setting, we propose to use uncertainty estimation techniques directly on the …
Bayesian InferenceDeep Reinforcement LearningOpen-Ended Question AnsweringOut-of-Distribution Detection+4SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning
The performance of offline reinforcement learning (RL) suffers from the limited size and quality of static datasets. Model-based offline RL addresses this issue by generating synthetic samples through a dynamics model to…
D4RLOffline RLReinforcement Learning (RL)Uncertainty Estimation Using Riemannian Model~Dynamics for Offline Reinforcement Learning
Model-based offline reinforcement learning approaches generally rely on bounds of model error. Estimating these bounds is usually achieved through uncertainty estimation methods. In this work, we combine parametric and n…
Autonomous Drivingcontinuous-controlContinuous ControlDecoder+4Query The Agent: Improving sample efficiency through epistemic uncertainty estimation
Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sampl…
reinforcement-learningReinforcement Learning (RL)Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of in…
continuous-controlContinuous Controlmodelreinforcement-learning+2