Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

2025-06-06 · Motoki Omura, Kazuki Ota, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada

For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continuous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overestimation bias. To address this, we propose an annealing approach that gradually transitions from the Bellman optimality operator to the Bellman operator, thereby accelerating learning while mitigating bias. Our method, combined with TD3 and SAC, significantly outperforms existing approaches across various locomotion and manipulation tasks, demonstrating improved performance and robustness to hyperparameters related to optimality.

📄 PDF Abstract BibTeX arXiv:2506.05968

Code (1)

motokiomura/annealed-q-learning 공식 구현 jax

Tasks

Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

SAC 설명 없음

Clipped Double Q-learning 설명 없음

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Parameterized Projected Bellman Operator

2023-12-20 · Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters 외

Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure wh…

Decision Making Reinforcement Learning (RL)Sequential Decision Making

Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problem

2026-04-08 · Hyukjun Yang, Han-Dong Lim, Donghwan Lee arxiv

The problem of solving Markov decision processes under function approximation remains a fundamental challenge, even under linear function approximation settings. A key difficulty arises from a geometric mismatch: while t…

Offline Deep Q* Estimation with Diffusion Models

2026-08-14 · Xiaohong Chen, Yuling Jiao, Lican Kang, Jerry Zhijian Yang 외 arxiv

In offline RL, estimating the optimal action-value function $Q^*$ can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and tr…

Offline RL

Linear $Q$-Learning Does Not Diverge: Convergence Rates to a Bounded Set

2025-01-31 · Xinyu Liu, Zixuan Xie, Shangtong Zhang

$Q$-learning is one of the most fundamental reinforcement learning algorithms. Previously, it is widely believed that $Q$-learning with linear function approximation (i.e., linear $Q$-learning) suffers from possible dive…

Q-Learning

Increasing the Action Gap: New Operators for Reinforcement Learning

2015-12-15 · Marc G. Bellemare, Georg Ostrovski, Arthur Guez, Philip S. Thomas 외

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistenc…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning+1