Off-Policy Actor-Critic with Emphatic Weightings

2021-11-16 · Eric Graves, Ehsan Imani, Raksha Kumaraswamy, Martha White

A variety of theoretically-sound policy gradient algorithms exist for the on-policy setting due to the policy gradient theorem, which provides a simplified form for the gradient. The off-policy setting, however, has been less clear due to the existence of multiple objectives and the lack of an explicit off-policy policy gradient theorem. In this work, we unify these objectives into one off-policy objective, and provide a policy gradient theorem for this unified objective. The derivation involves emphatic weightings and interest functions. We show multiple strategies to approximate the gradients, in an algorithm called Actor Critic with Emphatic weightings (ACE). We prove in a counterexample that previous (semi-gradient) off-policy actor-critic methods--particularly Off-Policy Actor-Critic (OffPAC) and Deterministic Policy Gradient (DPG)--converge to the wrong solution whereas ACE finds the optimal solution. We also highlight why these semi-gradient approaches can still perform well in practice, suggesting strategies for variance reduction in ACE. We empirically study several variants of ACE on two classic control environments and an image-based environment designed to illustrate the tradeoffs made by each gradient approximation. We find that by approximating the emphatic weightings directly, ACE performs as well as or better than OffPAC in all settings tested.

📄 PDF Abstract BibTeX arXiv:2111.08172

Code (1)

gravesec/actor-critic-with-emphatic-weightings 공식 구현

An Off-policy Policy Gradient Theorem Using Emphatic Weightings

2018-11-22 · NeurIPS 2018 12 · Ehsan Imani, Eric Graves, Martha White

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, …

Policy Gradient Methods Reinforcement Learning

Learning Expected Emphatic Traces for Deep RL

2021-07-12 · Ray Jiang, Shangtong Zhang, Veronica Chelu, Adam White 외

Off-policy sampling and experience replay are key for improving sample efficiency and scaling model-free temporal difference learning methods. When combined with function approximation, such as neural networks, this comb…

Generalized Off-Policy Actor-Critic

2019-03-27 · NeurIPS 2019 12 · Shangtong Zhang, Wendelin Boehmer, Shimon Whiteson

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excurs…

counterfactual MuJoCo reinforcement-learning Reinforcement Learning+1

Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation

2019-11-11 · ICML 2020 1 · Shangtong Zhang, Bo Liu, Hengshuai Yao, Shimon Whiteson

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained vi…

Vocal Bursts Valence Prediction

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

2019-03-15 · Wesley Suttle, Zhuoran Yang, Kaiqing Zhang, Zhaoran Wang 외

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)