How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?
Distributional reinforcement learning, which focuses on learning the entire return distribution instead of only its expectation in standard RL, has demonstrated remarkable success in enhancing performance. Despite these advancements, our comprehension of how the return distribution within distributional RL still remains limited. In this study, we investigate the optimization advantages of distributional RL by utilizing its extra return distribution knowledge over classical RL within the Neural Fitted Z-Iteration~(Neural FZI) framework. To begin with, we demonstrate that the distribution loss of distributional RL has desirable smoothness characteristics and hence enjoys stable gradients, which is in line with its tendency to promote optimization stability. Furthermore, the acceleration effect of distributional RL is revealed by decomposing the return distribution. It shows that distributional RL can perform favorably if the return distribution approximation is appropriate, measured by the variance of gradient estimates in each environment. Rigorous experiments validate the stable optimization behaviors of distributional RL and its acceleration effects compared to classical RL. Our research findings illuminate how the return distribution in distributional RL algorithms helps the optimization.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning
Distributional reinforcement learning aims to learn distribution of return under stochastic environments. Since the learned distribution of return contains rich information about the stochasticity of the environment, pre…
Atari GamesDescriptiveDistributional Reinforcement LearningEfficient Exploration+3Bayesian Distributional Policy Gradients
Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy perfo…
Atari GamesContrastive LearningDistributional Reinforcement LearningMuJoCo+1On solutions of the distributional Bellman equation
In distributional reinforcement learning not only expected returns but the complete return distributions of a policy are taken into account. The return distribution for a fixed policy is given as the solution of an assoc…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Risk-Sensitive Policy with Distributional Reinforcement Learning
Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into cons…
Decision MakingDistributional Reinforcement Learningreinforcement-learningReinforcement Learning+2Distributional Pareto-Optimal Multi-Objective Reinforcement Learning
Multi-objective reinforcement learning (MORL) has been proposed to learn control policies over multiple competing objectives with each possible preference over returns. However, current MORL algorithms fail to account fo…
Autonomous DrivingMulti-Objective Reinforcement Learningreinforcement-learningReinforcement Learning