Multi-Agent Trust Region Policy Optimization

2020-10-15 · Hepeng Li, Haibo He

We extend trust region policy optimization (TRPO) to multi-agent reinforcement learning (MARL) problems. We show that the policy update of TRPO can be transformed into a distributed consensus optimization problem for multi-agent cases. By making a series of approximations to the consensus optimization model, we propose a decentralized MARL algorithm, which we call multi-agent TRPO (MATRPO). This algorithm can optimize distributed policies based on local observations and private rewards. The agents do not need to know observations, rewards, policies or value/action-value functions of other agents. The agents only share a likelihood ratio with their neighbors during the training process. The algorithm is fully decentralized and privacy-preserving. Our experiments on two cooperative games demonstrate its robust performance on complicated MARL tasks.

📄 PDF Abstract BibTeX arXiv:2010.07916

Code (1)

hepengli/matrpo tf

Tasks

Multi-agent Reinforcement LearningPrivacy Preservingreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning

2026-05-09 · Chulabhaya Wijesundara, Andrea Baisero, Zhongheng Li, Gregory Castañón 외 arxiv

Centralized training with decentralized execution (CTDE) is a standard framework for cooperative multi-agent policy-gradient reinforcement learning, allowing agents to learn from joint information while acting from local…

Multi-agent Reinforcement Learning

A Game-Theoretic Approach to Multi-Agent Trust Region Optimization

2021-06-12 · Ying Wen, Hui Chen, Yaodong Yang, Zheng Tian 외

Trust region methods are widely applied in single-agent reinforcement learning problems due to their monotonic performance-improvement guarantee at every iteration. Nonetheless, when applied in multi-agent settings, the …

Atari Games MuJoCo Multi-agent Reinforcement Learning reinforcement-learning+2

Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach

2025-08-14 · Chak Lam Shek, Guangyao Shi, Pratap Tokekar arxiv

Multi-agent reinforcement learning (MARL) requires coordinated and stable policy updates among interacting agents. Heterogeneous-Agent Trust Region Policy Optimization (HATRPO) enforces per-agent trust region constraints…

Multi-agent Reinforcement Learning

An Analytical Update Rule for General Policy Optimization

2021-12-03 · Hepeng Li, Nicholas Clavette, Haibo He

We present an analytical policy update rule that is independent of parametric function approximators. The policy update rule is suitable for optimizing general stochastic policies and has a monotonic improvement guarante…

reinforcement-learning Reinforcement Learning (RL)

Multi-Agent Trust Region Learning

2021-01-01 · Ying Wen, Hui Chen, Yaodong Yang, Zheng Tian 외

Trust-region methods are widely used in single-agent reinforcement learning. One advantage is that they guarantee a lower bound of monotonic payoff improvement for policy optimization at each iteration. Nonetheless, whe…

Atari Games MuJoCo Multi-agent Reinforcement Learning Q-Learning+2