Bayesian Bellman Operators

2021-06-09 · NeurIPS 2021 12 · Matthew Fellows, Kristian Hartikainen, Shimon Whiteson

We introduce a novel perspective on Bayesian reinforcement learning (RL); whereas existing approaches infer a posterior over the transition distribution or Q-function, we characterise the uncertainty in the Bellman operator. Our Bayesian Bellman operator (BBO) framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions. In this paper, we use BBO to provide a rigorous theoretical analysis of model-free Bayesian RL to better understand its relationshipto established frequentist RL methodologies. We prove that Bayesian solutions are consistent with frequentist RL solutions, even when approximate inference isused, and derive conditions for which convergence properties hold. Empirically, we demonstrate that algorithms derived from the BBO framework have sophisticated deep exploration properties that enable them to solve continuous control tasks at which state-of-the-art regularised actor-critic algorithms fail catastrophically

📄 PDF Abstract BibTeX arXiv:2106.05012

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control Reinforcement Learning (RL)

Bellman operator convergence enhancements in reinforcement learning algorithms

2025-05-20 · David Krame Kadurha, Domini Jocema Leko Moutouo, Yae Ulrich Gaba

This paper reviews the topological groundwork for the study of reinforcement learning (RL) by focusing on the structure of state, action, and policy spaces. We begin by recalling key mathematical concepts such as complet…

Acrobot Decision Making reinforcement-learning Reinforcement Learning+1

To bootstrap or to rollout? An optimal and adaptive interpolation

2024-11-14 · Wenlong Mou, Jian Qian

Bootstrapping and rollout are two fundamental principles for value function estimation in reinforcement learning (RL). We introduce a novel class of Bellman operators, called subgraph Bellman operators, that interpolate …

Reinforcement Learning (RL)

Increasing the Action Gap: New Operators for Reinforcement Learning

2015-12-15 · Marc G. Bellemare, Georg Ostrovski, Arthur Guez, Philip S. Thomas 외

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistenc…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1

A General Family of Robust Stochastic Operators for Reinforcement Learning

2018-05-21 · Yingdong Lu, Mark S. Squillante, Chai Wah Wu

We consider a new family of operators for reinforcement learning with the goal of alleviating the negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are establish…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Family of Robust Stochastic Operators for Reinforcement Learning

2019-12-01 · NeurIPS 2019 12 · Yingdong Lu, Mark Squillante, Chai Wah Wu

We consider a new family of stochastic operators for reinforcement learning with the goal of alleviating negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are es…