Order Matters: Agent-by-agent Policy Optimization

2023-02-13 · Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, Weinan Zhang

While multi-agent trust region algorithms have achieved great success empirically in solving coordination tasks, most of them, however, suffer from a non-stationarity problem since agents update their policies simultaneously. In contrast, a sequential scheme that updates policies agent-by-agent provides another perspective and shows strong performance. However, sample inefficiency and lack of monotonic improvement guarantees for each agent are still the two significant challenges for the sequential scheme. In this paper, we propose the \textbf{A}gent-by-\textbf{a}gent \textbf{P}olicy \textbf{O}ptimization (A2PO) algorithm to improve the sample efficiency and retain the guarantees of monotonic improvement for each agent during training. We justify the tightness of the monotonic improvement bound compared with other trust region algorithms. From the perspective of sequentially updating agents, we further consider the effect of agent updating order and extend the theory of non-stationarity into the sequential update scheme. To evaluate A2PO, we conduct a comprehensive empirical study on four benchmarks: StarCraftII, Multi-agent MuJoCo, Multi-agent Particle Environment, and Google Research Football full game scenarios. A2PO consistently outperforms strong baselines.

📄 PDF Abstract BibTeX arXiv:2302.06205

Code (1)

xihuai18/A2PO-ICLR2023 공식 구현 pytorch

Tasks

MuJoCo

Methods 이 논문이 사용한 방법론

((Reservation@Faqs))How do I cancel a reservation on Expedia? How do I cancel a reservation on Expedia? +1^888^829^0881° oR +1^888^829^0881 – Need to cancel your Expedia reservation quickly and without hassle? This step-by-step guide…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Six Ways To Communicate To Someone At Expedia Via Phone And Email's. To communicate or get human at Expedia, the quickest option is typically to call their customer service at ＋１－８８８－８２９－０８８１　ｏｒ　＋１（８０５）　３３０　（４０５６）. You can also use the live chat…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

TTUR The Two Time-scale Update Rule (TTUR) is an update rule for generative adversarial networks trained with stochastic gradient descent. TTUR has an individual learning rate for…

Projection Discriminator A Projection Discriminator is a type of discriminator for generative adversarial networks. It is motivated by a probabilistic model in which the distribution of the…

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

2020-05-25 · Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras 외

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Implementation Matters in Deep RL: A Case Study on PPO and TRPO

2020-05-01 · ICLR 2020 1 · Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras 외

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms, Proximal Policy Optimization and Trust Region Policy Optimization. We investigate the conseque…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Multi-Agent First Order Constrained Optimization in Policy Space

2023-09-21 · NeurIPS 2023 11

In the realm of multi-agent reinforcement learning (MARL), achieving high performance is crucial for a successful multi-agent system. Meanwhile, the ability to avoid unsafe actions is becoming an urgent and imperative pr…

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

2020-06-18 · Yan Zhang, Michael M. Zavlanos

In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

First Order Constrained Optimization in Policy Space

2020-02-16 · NeurIPS 2020 12 · Yiming Zhang, Quan Vuong, Keith W. Ross

In reinforcement learning, an agent attempts to learn high-performing behaviors through interacting with the environment, such behaviors are often quantified in the form of a reward function. However some aspects of beha…

Reinforcement Learning