Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning

2021-11-25 · Xiaoxiao Zhao, Jinlong Lei, Li Li, Jie Chen

This paper studies a distributed policy gradient in collaborative multi-agent reinforcement learning (MARL), where agents over a communication network aim to find the optimal policy to maximize the average of all agents' local returns. Due to the non-concave performance function of policy gradient, the existing distributed stochastic optimization methods for convex problems cannot be directly used for policy gradient in MARL. This paper proposes a distributed policy gradient with variance reduction and gradient tracking to address the high variances of policy gradient, and utilizes importance weight to solve the {distribution shift} problem in the sampling process. We then provide an upper bound on the mean-squared stationary gap, which depends on the number of iterations, the mini-batch size, the epoch size, the problem parameters, and the network topology. We further establish the sample and communication complexity to obtain an $\epsilon$-approximate stationary point. Numerical experiments are performed to validate the effectiveness of the proposed algorithm.

📄 PDF Abstract BibTeX arXiv:2111.12961

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)Stochastic Optimization

SoftTreeMax: Exponential Variance Reduction in Policy Gradient via Tree Search

2023-01-30 · Gal Dalal, Assaf Hallak, Gugan Thoppe, Shie Mannor 외

Despite the popularity of policy gradient methods, they are known to suffer from large variance and high sample complexity. To mitigate this, we introduce SoftTreeMax -- a generalization of softmax that takes planning in…

GPU Policy Gradient Methods

Variance Reduction for Distributed Stochastic Gradient Descent

2015-12-05 · Soham De, Gavin Taylor, Tom Goldstein

Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD …

Stochastic Optimization

From Importance Sampling to Doubly Robust Policy Gradient

2019-10-20 · ICML 2020 1 · Jiawei Huang, Nan Jiang

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-p…

Off-policy evaluation

Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

2018-03-20 · ICLR 2018 1 · Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar 외

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons …

Deep Reinforcement Learning Policy Gradient Methods reinforcement-learning Reinforcement Learning+1

Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy

2019-03-13 · Yunhao Tang, Mingzhang Yin, Mingyuan Zhou

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-…