Distributional Reinforcement Learning for Efficient Exploration
In distributional reinforcement learning (RL), the estimated distribution of value function models both the parametric and intrinsic uncertainties. We propose a novel and efficient exploration method for deep RL that has two components. The first is a decaying schedule to suppress the intrinsic uncertainty. The second is an exploration bonus calculated from the upper quantiles of the learned distribution. In Atari 2600 games, our method outperforms QR-DQN in 12 out of 14 hard games (achieving 483 \% average gain across 49 games in cumulative rewards over QR-DQN with a big win in Venture). We also compared our algorithm with QR-DQN in a challenging 3D driving simulator (CARLA). Results show that our algorithm achieves near-optimal safety rewards twice faster than QRDQN.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesDistributional Reinforcement LearningEfficient Explorationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Exploration by Distributional Reinforcement Learning
We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies m…
Deep Reinforcement LearningDistributional Reinforcement LearningEfficient Explorationreinforcement-learning+2Risk Perspective Exploration in Distributional Reinforcement Learning
Distributional reinforcement learning demonstrates state-of-the-art performance in continuous and discrete control settings with the features of variance and risk, which can be used to explore. However, the exploration m…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic explorat…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningExploration with Multi-Sample Target Values for Distributional Reinforcement Learning
Distributional reinforcement learning (RL) aims to learn a value-network that predicts the full distribution of the returns for a given state, often modeled via a quantile-based critic. This approach has been successfull…
continuous-controlContinuous ControlDistributional Reinforcement LearningHumanoid Control+3The Benefits of Being Categorical Distributional: Uncertainty-aware Regularized Exploration in Reinforcement Learning
The remarkable empirical performance of distributional reinforcement learning (RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distribut…
Atari GamesAttributeDistributional Reinforcement Learningreinforcement-learning+1