Increasing the Action Gap: New Operators for Reinforcement Learning

2015-12-15 · Marc G. Bellemare, Georg Ostrovski, Arthur Guez, Philip S. Thomas, Rémi Munos

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird's advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.

📄 PDF Abstract BibTeX arXiv:1512.04860

Code (2)

chainer/chainerrl pytorch

janhuenermann/neurojs tf

Tasks

Atari GamesQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Exploring grid topology reconfiguration using a simple deep reinforcement learning approach

2020-11-26 · Medha Subramanian, Jan Viebahn, Simon H. Tindemans, Benjamin Donnot 외

System operators are faced with increasingly volatile operating conditions. In order to manage system reliability in a cost-effective manner, control room operators are turning to computerised decision support tools base…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)Time Series+1

A General Family of Robust Stochastic Operators for Reinforcement Learning

2018-05-21 · Yingdong Lu, Mark S. Squillante, Chai Wah Wu

We consider a new family of operators for reinforcement learning with the goal of alleviating the negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are establish…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Family of Robust Stochastic Operators for Reinforcement Learning

2019-12-01 · NeurIPS 2019 12 · Yingdong Lu, Mark Squillante, Chai Wah Wu

We consider a new family of stochastic operators for reinforcement learning with the goal of alleviating negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are es…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Meta-operators for Enabling Parallel Planning Using Deep Reinforcement Learning

2024-03-13 · Ángel Aso-Mollar, Eva Onaindia

There is a growing interest in the application of Reinforcement Learning (RL) techniques to AI planning with the aim to come up with general policies. Typically, the mapping of the transition model of AI planning to the …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Bellman operator convergence enhancements in reinforcement learning algorithms

2025-05-20 · David Krame Kadurha, Domini Jocema Leko Moutouo, Yae Ulrich Gaba

This paper reviews the topological groundwork for the study of reinforcement learning (RL) by focusing on the structure of state, action, and policy spaces. We begin by recalling key mathematical concepts such as complet…

Acrobot Decision Making reinforcement-learning Reinforcement Learning+1