Off-Policy Actor-Critic

2012-05-22 · Thomas Degris, Martha White, Richard S. Sutton

This paper presents the first actor-critic algorithm for off-policy reinforcement learning. Our algorithm is online and incremental, and its per-time-step complexity scales linearly with the number of learned weights. Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning. Off-policy techniques, such as Greedy-GQ, enable a target policy to be learned while following and obtaining data from another (behavior) policy. For many problems, however, actor-critic methods are more practical than action value methods (like Greedy-GQ) because they explicitly represent the policy; consequently, the policy can be stochastic and utilize a large action space. In this paper, we illustrate how to practically combine the generality and learning potential of off-policy learning with the flexibility in action selection given by actor-critic methods. We derive an incremental, linear time and space complexity algorithm that includes eligibility traces, prove convergence under assumptions similar to previous off-policy algorithms, and empirically show better or comparable performance to existing algorithms on standard reinforcement-learning benchmark problems.

📄 PDF Abstract BibTeX arXiv:1205.4839

Code (1)

DevSlem/OfflineRL/blob/main/chap1_introduction.ipynb

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Actor-Advisor: Policy Gradient With Off-Policy Advice

2019-02-07 · Hélène Plisnier, Denis Steckelmacher, Diederik M. Roijers, Ann Nowé

Actor-critic algorithms learn an explicit policy (actor), and an accompanying value function (critic). The actor performs actions in the environment, while the critic evaluates the actor's current policy. However, despit…

Transfer Learning

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

2019-12-11 · Riashat Islam, Raihan Seraj, Samin Yeasar Arnob, Doina Precup

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+2

Guide Actor-Critic for Continuous Control

2017-05-22 · ICLR 2018 1 · Voot Tangkaratt, Abbas Abdolmaleki, Masashi Sugiyama

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

2019-03-11 · Denis Steckelmacher, Hélène Plisnier, Diederik M. Roijers, Ann Nowé

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling

Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy

2020-08-02 · ICLR 2021 1 · Zuyue Fu, Zhuoran Yang, Zhaoran Wang

We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms. While most existing works on actor-critic employ bi-level or two-timescale upd…