Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning algorithm that selects actions by randomizing risk criterion to avoid one-sided tendency on risk. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure and prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning
Distributional reinforcement learning aims to learn distribution of return under stochastic environments. Since the learned distribution of return contains rich information about the stochasticity of the environment, pre…
Atari GamesDescriptiveDistributional Reinforcement LearningEfficient Exploration+3DROP: Distributional and Regular Optimism and Pessimism for Reinforcement Learning
In reinforcement learning (RL), temporal difference (TD) error is known to be related to the firing rate of dopamine neurons. It has been observed that each dopamine neuron does not behave uniformly, but each responds to…
Ensemble Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds
We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective…
Computational EfficiencyDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+1Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy
While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, rel…
Reinforcement LearningOptimism as Risk-Seeking in Multi-Agent Reinforcement Learning
Risk sensitivity has become a central theme in reinforcement learning (RL), where convex risk measures and robust formulations provide principled ways to model preferences beyond expected return. Recent extensions to mul…
Multi-agent Reinforcement Learning