Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning

2018-12-07 · Tianyi Chen, Kaiqing Zhang, Georgios B. Giannakis, Tamer Başar

This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.

📄 PDF Abstract BibTeX arXiv:1812.03239

Code (0)

등록된 구현이 없습니다.

Tasks

Distributed ComputingMulti-agent Reinforcement LearningPolicy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning

2021-11-25 · Xiaoxiao Zhao, Jinlong Lei, Li Li, Jie Chen

This paper studies a distributed policy gradient in collaborative multi-agent reinforcement learning (MARL), where agents over a communication network aim to find the optimal policy to maximize the average of all agents'…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)Stochastic Optimization

Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning

2025-09-29 · Alexander Tyurin, Andrei Spiridonov, Varvara Rudenko arxiv

We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achi…

Reinforcement Learning

Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach

2019-12-19 · L4DC 2020 6 · Ying-Ying Li, Yujie Tang, Runyu Zhang, Na Li

This paper considers a distributed reinforcement learning problem for decentralized linear quadratic control with partial state observations and local costs. We propose a Zero-Order Distributed Policy Optimization algori…

Reinforcement Learning Reinforcement Learning (RL)

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

2022-01-10 · Gangshan Jing, He Bai, Jemin George, Aranya Chakrabortty 외

Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for p…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

2020-06-18 · Yan Zhang, Michael M. Zavlanos

In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of …