Value-Distributional Model-Based Reinforcement Learning
Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks. We study the problem from a model-based Bayesian reinforcement learning perspective, where the goal is to learn the posterior distribution over value functions induced by parameter (epistemic) uncertainty of the Markov decision process. Previous work restricts the analysis to a few moments of the distribution over values or imposes a particular distribution shape, e.g., Gaussians. Inspired by distributional reinforcement learning, we introduce a Bellman operator whose fixed-point is the value distribution function. Based on our theory, we propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function. We combine EQR with soft actor-critic (SAC) for policy optimization with an arbitrary differentiable objective function of the learned value distribution. Evaluation across several continuous-control tasks shows performance benefits with respect to both model-based and model-free algorithms. The code is available at https://github.com/boschresearch/dist-mbrl.
Code (1)
Tasks
continuous-controlContinuous ControlDecision MakingDistributional Reinforcement LearningmodelModel-based Reinforcement Learningquantile regressionreinforcement-learningReinforcement LearningSequential Decision MakingSimilar Papers 제목 키워드 기반
An Analysis of Categorical Distributional Reinforcement Learning
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performan…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Distributional Reinforcement Learning via the Cramér Distance
This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cramér-based Distributional Sof…
Reinforcement LearningExploration with Multi-Sample Target Values for Distributional Reinforcement Learning
Distributional reinforcement learning (RL) aims to learn a value-network that predicts the full distribution of the returns for a given state, often modeled via a quantile-based critic. This approach has been successfull…
continuous-controlContinuous ControlDistributional Reinforcement LearningHumanoid Control+3Distributional Reinforcement Learning with Quantile Regression
In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce ra…
Atari GamesDistributional Reinforcement Learningquantile regressionregression+3A Distributional Perspective on Reinforcement Learning
In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforceme…
Atari Gamesreinforcement-learningReinforcement LearningReinforcement Learning (RL)