Revisiting the Softmax Bellman Operator: New Benefits and New Perspective

2018-12-02 · Zhao Song, Ronald E. Parr, Lawrence Carin

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on exploration, the softmax Bellman operator when combined with Deep Q-learning, leads to Q-functions with superior policies in practice, even outperforming its double Q-learning counterpart. To better understand how and why this occurs, we revisit theoretical properties of the softmax Bellman operator, and prove that $(i)$ it converges to the standard Bellman operator exponentially fast in the inverse temperature parameter, and $(ii)$ the distance of its Q function from the optimal one can be bounded. These alone do not explain its superior performance, so we also show that the softmax operator can reduce the overestimation error, which may give some insight into why a sub-optimal operator leads to better performance in the presence of value function approximation. A comparison among different Bellman operators is then presented, showing the trade-offs when selecting them.

📄 PDF Abstract BibTeX arXiv:1812.00456

Code (2)

zhao-song/Softmax-DQN 공식 구현

Wadaboa/flatland-challenge pytorch

Tasks

Atari GamesQ-LearningReinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Parameterized Projected Bellman Operator

2023-12-20 · Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters 외

Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure wh…

Decision Making Reinforcement Learning (RL)Sequential Decision Making

Bayesian Bellman Operators

2021-06-09 · NeurIPS 2021 12 · Matthew Fellows, Kristian Hartikainen, Shimon Whiteson

We introduce a novel perspective on Bayesian reinforcement learning (RL); whereas existing approaches infer a posterior over the transition distribution or Q-function, we characterise the uncertainty in the Bellman opera…

continuous-control Continuous Control Reinforcement Learning (RL)

Reinforcement Learning with Dynamic Boltzmann Softmax Updates

2019-03-14 · Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen 외

Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the n…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1

Linear $Q$-Learning Does Not Diverge: Convergence Rates to a Bounded Set

2025-01-31 · Xinyu Liu, Zixuan Xie, Shangtong Zhang

$Q$-learning is one of the most fundamental reinforcement learning algorithms. Previously, it is widely believed that $Q$-learning with linear function approximation (i.e., linear $Q$-learning) suffers from possible dive…

Q-Learning

Softmax Deep Double Deterministic Policy Gradients

2020-10-19 · NeurIPS 2020 12 · Ling Pan, Qingpeng Cai, Longbo Huang

A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Althou…

continuous-control Continuous Control