Is the Bellman residual a bad proxy?

2016-06-24 · NeurIPS 2017 12 · Matthieu Geist, Bilal Piot, Olivier Pietquin

This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximize the mean value, and derive a method that minimizes the residual $\|T_* v_\pi - v_\pi\|_{1,\nu}$ over policies. A theoretical analysis shows how good this proxy is to policy optimization, and notably that it is better than its value-based counterpart. We also propose experiments on randomly generated generic Markov decision processes, specifically designed for studying the influence of the involved concentrability coefficient. They show that the Bellman residual is generally a bad proxy to policy optimization and that directly maximizing the mean value is much better, despite the current lack of deep theoretical analysis. This might seem obvious, as directly addressing the problem of interest is usually better, but given the prevalence of (projected) Bellman residual minimization in value-based reinforcement learning, we believe that this question is worth to be considered.

📄 PDF Abstract BibTeX arXiv:1606.07636

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence

2026-01-26 · Donghwan Lee, Hyukjun Yang arxiv

Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to…

Reinforcement Learning

Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problem

2026-04-08 · Hyukjun Yang, Han-Dong Lim, Donghwan Lee arxiv

The problem of solving Markov decision processes under function approximation remains a fundamental challenge, even under linear function approximation settings. A key difficulty arises from a geometric mismatch: while t…

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error

2022-01-28 · Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum 외

In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value function over all state-action pairs, we find…

Value prediction

Why Should I Trust You, Bellman? Evaluating the Bellman Objective with Off-Policy Data

2021-09-29 · Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum 외

In this work, we analyze the effectiveness of the Bellman equation as a proxy objective for value prediction accuracy in off-policy evaluation. While the Bellman equation is uniquely solved by the true value function ove…

Deep Reinforcement Learning Off-policy evaluation Value prediction

Robust Value Function Approximation Using Bilinear Programming

2009-12-01 · NeurIPS 2009 12 · Marek Petrik, Shlomo Zilberstein

Existing value function approximation methods have been successfully used in many applications, but they often lack useful a priori error bounds. We propose approximate bilinear programming, a new formulation of value fu…