Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning
Distributional reinforcement learning aims to learn distribution of return under stochastic environments. Since the learned distribution of return contains rich information about the stochasticity of the environment, previous studies have relied on descriptive statistics, such as standard deviation, for optimism in face of uncertainty. These prior works are divided into risk-seeking or averse methods, which can be considered as having a one-sided tendency on risk. Unexpectedly, such approaches hinder convergence. In this paper, we propose a novel distributional reinforcement learning that explores by randomizing the risk criterion to reach a risk-neutral optimal policy. First, we provide a perturbed distributional Bellman optimality operator by distorting the risk measure in action selection. Second, we prove the convergence and optimality of the proposed method by using weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and converges to an optimal return distribution. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari games.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari GamesDescriptiveDistributional Reinforcement LearningEfficient Explorationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Exploration by Distributional Reinforcement Learning
We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies m…
Deep Reinforcement LearningDistributional Reinforcement LearningEfficient Explorationreinforcement-learning+2Exploration with Multi-Sample Target Values for Distributional Reinforcement Learning
Distributional reinforcement learning (RL) aims to learn a value-network that predicts the full distribution of the returns for a given state, often modeled via a quantile-based critic. This approach has been successfull…
continuous-controlContinuous ControlDistributional Reinforcement LearningHumanoid Control+3The Benefits of Being Categorical Distributional: Uncertainty-aware Regularized Exploration in Reinforcement Learning
The remarkable empirical performance of distributional reinforcement learning (RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distribut…
Atari GamesAttributeDistributional Reinforcement Learningreinforcement-learning+1Risk Perspective Exploration in Distributional Reinforcement Learning
Distributional reinforcement learning demonstrates state-of-the-art performance in continuous and discrete control settings with the features of variance and risk, which can be used to explore. However, the exploration m…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic explorat…
Distributional Reinforcement Learningreinforcement-learningReinforcement Learning