Variance Penalized On-Policy and Off-Policy Actor-Critic

2021-02-03 · Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, Doina Precup

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an algorithm. In this paper, we propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Previous work uses the second moment of return to estimate the variance indirectly. Instead, we use a much simpler recently proposed direct variance estimator which updates the estimates incrementally using temporal difference methods. Using the variance-penalized criterion, we guarantee the convergence of our algorithm to locally optimal policies for finite state action Markov decision processes. We demonstrate the utility of our algorithm in tabular and continuous MuJoCo domains. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.

📄 PDF Abstract BibTeX arXiv:2102.01985

Code (1)

arushi12130/VariancePenalizedActorCritic 공식 구현 pytorch

Tasks

MuJoCo

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

2019-12-11 · Riashat Islam, Raihan Seraj, Samin Yeasar Arnob, Doina Precup

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+2

Relative Importance Sampling for off-Policy Actor-Critic in Deep Reinforcement Learning

2018-10-30 · Mahammad Humayoo, Gengzhong Zheng, Xiaoqing Dong, Liming Miao 외

Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy ($\pi$) and the behavior policy (b…

Deep Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning+1

Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective

2021-10-06 · Ting-Han Fan, Peter J. Ramadge

Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on the distribution of transitions decompos…

Actor-Critic with Active Importance Sampling

2026-05-08 · Majid Molaei, Gabor Paczolay, Matteo Papini, Alberto Maria Metelli 외 arxiv

This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm, an extension of the Actor-Critic framework for reducing variance in policy gradient estimation. AISAC optimizes the behavior policy to …

Reinforcement Learning

Learning to Optimize Job Shop Scheduling Under Structural Uncertainty

2026-01-29 · Rui Zhang, Jianwei Niu, Xuefeng Liu, Shaojie Tang 외 arxiv

The Job-Shop Scheduling Problem (JSSP), under various forms of manufacturing uncertainty, has recently attracted considerable research attention. Most existing studies focus on parameter uncertainty, such as variable pro…