Operator Shifting for Model-based Policy Evaluation

2021-10-25 · Xun Tang, Lexing Ying, Yuhua Zhu

In model-based reinforcement learning, the transition matrix and reward vector are often estimated from random samples subject to noise. Even if the estimated model is an unbiased estimate of the true underlying model, the value function computed from the estimated model is biased. We introduce an operator shifting method for reducing the error introduced by the estimated model. When the error is in the residual norm, we prove that the shifting factor is always positive and upper bounded by $1+O\left(1/n\right)$, where $n$ is the number of samples used in learning each row of the transition matrix. We also propose a practical numerical algorithm for implementing the operator shifting.

📄 PDF Abstract BibTeX arXiv:2110.12658

Code (0)

등록된 구현이 없습니다.

Tasks

modelModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning (RL)

Marginalized Operators for Off-policy Reinforcement Learning

2022-03-30 · Yunhao Tang, Mark Rowland, Rémi Munos, Michal Valko

In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as spe…

Off-policy evaluation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On the Fundamental Limitations of Dual Static CVaR Decompositions in Markov Decision Processes

2025-07-18 · Mathieu Godbout, Audrey Durand arxiv

It was recently shown that dynamic programming (DP) methods for finding static CVaR-optimal policies in Markov Decision Processes (MDPs) can fail when based on the dual formulation, yet the root cause of this failure rem…

Optimal variance-reduced stochastic approximation in Banach spaces

2022-01-21 · Wenlong Mou, Koulik Khamaru, Martin J. Wainwright, Peter L. Bartlett 외

We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. Focusing on a stochastic query model that provides noisy evaluations of the operator, we analyze a varianc…

Q-Learning

Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation

2026-02-16 · Olivier Jeunen, Shashank Gupta arxiv

Off-policy evaluation (OPE) is essential for assessing ranking and recommendation systems without costly online interventions. Self-Normalised Inverse Propensity Scoring (SNIPS) is a standard tool for variance reduction …

Recommendation Systems

Emphatic TD Bellman Operator is a Contraction

2015-08-14 · Assaf Hallak, Aviv Tamar, Shie Mannor

Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that u…

Off-policy evaluation