Coordinated Proximal Policy Optimization
We present Coordinated Proximal Policy Optimization (CoPPO), an algorithm that extends the original Proximal Policy Optimization (PPO) to the multi-agent setting. The key idea lies in the coordinated adaptation of step size during the policy update process among multiple agents. We prove the monotonicity of policy improvement when optimizing a theoretically-grounded joint objective, and derive a simplified optimization objective based on a set of approximations. We then interpret that such an objective in CoPPO can achieve dynamic credit assignment among agents, thereby alleviating the high variance issue during the concurrent update of agent policies. Finally, we demonstrate that CoPPO outperforms several strong baselines and is competitive with the latest multi-agent PPO method (i.e. MAPPO) under typical multi-agent settings, including cooperative matrix games and the StarCraft II micromanagement tasks.
Code (1)
Tasks
StarcraftStarcraft IIMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Heterogeneous Multi-Agent Proximal Policy Optimization for Power Distribution System Restoration
Restoring power distribution systems (PDSs) after large-scale outages requires sequential switching actions that reconfigure feeder topology and coordinate distributed energy resources (DERs) under nonlinear constraints,…
Reinforcement LearningProximal Policy Optimization for Integrated Sensing and Communication in mmWave Systems
In wireless communication systems, mmWave beam tracking is a critical task that affects both sensing and communications, as it is related to the knowledge of the wireless channel. We consider a setup in which a Base Stat…
Integrated sensing and communicationRiemannian Proximal Policy Optimization
In this paper, We propose a general Riemannian proximal optimization algorithm with guaranteed convergence to solve Markov decision process (MDP) problems. To model policy functions in MDP, we employ Gaussian mixture mod…
Decentralized Inexact Proximal Gradient Method With Network-Independent Stepsizes for Convex Composite Optimization
This paper proposes a novel CTA (Combine-Then-Adapt)-based decentralized algorithm for solving convex composite optimization problems over undirected and connected networks. The local loss function in these problems cont…
Proximal Policy Optimization with Mixed Distributed Training
Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algori…
Deep Reinforcement LearningReinforcement Learning