Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective

2021-10-06 · Ting-Han Fan, Peter J. Ramadge

Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling. By comparing the magnitude of bias and variance, we explain the success of the Emphasizing Recent Experience sampling and 1/age weighted sampling. Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.

📄 PDF Abstract BibTeX arXiv:2110.02421

Code (1)

sunfex/weighted-sac 공식 구현 pytorch

Actor-Critic with Active Importance Sampling

2026-05-08 · Majid Molaei, Gabor Paczolay, Matteo Papini, Alberto Maria Metelli 외 arxiv

This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm, an extension of the Actor-Critic framework for reducing variance in policy gradient estimation. AISAC optimizes the behavior policy to …

Reinforcement Learning

Refined Analysis of Entropy-Regularized Actor-Critic

2026-05-23 · Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines arxiv

In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reductio…

Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm

2021-02-18 · Sajad Khodadadian, Zaiwei Chen, Siva Theja Maguluri

In this paper, we provide finite-sample convergence guarantees for an off-policy variant of the natural actor-critic (NAC) algorithm based on Importance Sampling. In particular, we show that the algorithm converges to a …

Unbiased Asymmetric Reinforcement Learning under Partial Observability

2021-05-25 · Andrea Baisero, Christopher Amato

In partially observable reinforcement learning, offline training gives access to latent information which is not available during online training and/or execution, such as the system state. Asymmetric actor-critic method…

Partially Observable Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Interpretable Meta-Reinforcement Learning with Actor-Critic Method

2021-01-01 · Xingyuan Liang, Xu-Ying Liu

Meta-reinforcement learning (meta-RL) algorithms have successfully trained agent systems to perform well on different tasks within only few updates. However, in gradient-based meta-RL algorithms, the Q-function at adapta…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)