Bayesian Distributional Policy Gradients

2021-03-20 · Luchen Li, A. Aldo Faisal

Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy performance, which may be beneficial for trading off exploration and exploitation and policy learning in general. Previous works in distributional RL focused mainly on computing the state-action-return distributions, here we model the state-return distributions. This enables us to translate successful conventional RL algorithms that are based on state values into distributional RL. We formulate the distributional Bellman operation as an inference-based auto-encoding process that minimises Wasserstein metrics between target/model return distributions. The proposed algorithm, BDPG (Bayesian Distributional Policy Gradients), uses adversarial training in joint-contrastive learning to estimate a variational posterior from the returns. Moreover, we can now interpret the return prediction uncertainty as an information gain, which allows to obtain a new curiosity measure that helps BDPG steer exploration actively and efficiently. We demonstrate in a suite of Atari 2600 games and MuJoCo tasks, including well known hard-exploration challenges, how BDPG learns generally faster and with higher asymptotic performance than reference distributional RL algorithms.

📄 PDF Abstract BibTeX arXiv:2103.11265

Code (0)

등록된 구현이 없습니다.

Tasks

Atari Games Contrastive LearningDistributional Reinforcement LearningMuJoCoReinforcement Learning (RL)

Bag of Policies for Distributional Deep Exploration

2023-08-03 · Asen Nachkov, Luchen Li, Giulia Luise, Filippo Valdettaro 외

Efficient exploration in complex environments remains a major challenge for reinforcement learning (RL). Compared to previous Thompson sampling-inspired mechanisms that enable temporally extended exploration, i.e., deep …

Atari Games Efficient Exploration Reinforcement Learning (RL)Thompson Sampling

Sample-based Distributional Policy Gradient

2020-01-08 · Rahul Singh, Keuntaek Lee, Yongxin Chen

Distributional reinforcement learning (DRL) is a recent reinforcement learning framework whose success has been supported by various empirical studies. It relies on the key idea of replacing the expected return with the …

Distributional Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning+1

Distributed Distributional Deterministic Policy Gradients

2018-04-23 · ICLR 2018 1 · Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney 외

This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order t…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

Bayesian Policy Gradients via Alpha Divergence Dropout Inference

2017-12-06 · Peter Henderson, Thang Doan, Riashat Islam, David Meger

Policy gradient methods have had great success in solving continuous control tasks, yet the stochastic nature of such problems makes deterministic value estimation difficult. We propose an approach which instead estimate…

continuous-control Continuous Control MuJoCo Policy Gradient Methods

Duality and Policy Evaluation in Distributionally Robust Bayesian Diffusion Control

2025-06-24 · Jose Blanchet, Jiayi Cheng, Hao liu, Yang Liu

We consider a Bayesian diffusion control problem of expected terminal utility maximization. The controller imposes a prior distribution on the unknown drift of an underlying diffusion. The Bayesian optimal control, track…