Multi-Agent Trust Region Learning
Trust-region methods are widely used in single-agent reinforcement learning. One advantage is that they guarantee a lower bound of monotonic payoff improvement for policy optimization at each iteration. Nonetheless, when applied in multi-agent settings, such guarantee is lost because an agent's payoff is also determined by other agents' adaptive behaviors. In fact, measuring agents' payoff improvements in multi-agent reinforcement learning (MARL) scenarios is still challenging. Although game-theoretical solution concepts such as Nash equilibrium can be applied, the algorithm (e.g., Nash-Q learning) suffers from poor scalability beyond two-player discrete games. To mitigate the above measurability and tractability issues, in this paper, we propose Multi-Agent Trust Region Learning (MATRL) method. MATRL augments the single-agent trust-region optimization process with the multi-agent solution concept of stable fixed point that is computed at the policy-space meta-game level. When multiple agents learn simultaneously, stable fixed points at the meta-game level can effectively measure agents' payoff improvements, and, importantly, a meta-game representation enjoys better scalability for multi-player games. We derive the lower bound of agents' payoff improvements for MATRL methods, and also prove the convergence of our method on the meta-game fixed points. We evaluate the MATRL method on both discrete and continuous multi-player general-sum games; results suggest that MATRL significantly outperforms strong MARL baselines on grid worlds, multi-agent MuJoCo, and Atari games.
Code (1)
Tasks
Atari GamesMuJoCoMulti-agent Reinforcement LearningQ-Learningreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
Trust region methods are widely applied in single-agent reinforcement learning problems due to their monotonic performance-improvement guarantee at every iteration. Nonetheless, when applied in multi-agent settings, the …
Atari GamesMuJoCoMulti-agent Reinforcement Learningreinforcement-learning+2Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
Trust region methods rigorously enabled reinforcement learning (RL) agents to learn monotonically improving policies, leading to superior performance on a variety of tasks. Unfortunately, when it comes to multi-agent rei…
LEMMAMuJoCoMulti-agent Reinforcement Learningreinforcement-learning+2Trust Region Bounds for Decentralized PPO Under Non-stationarity
We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides …
Multi-agent Reinforcement LearningRethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
Centralized training with decentralized execution (CTDE) is a standard framework for cooperative multi-agent policy-gradient reinforcement learning, allowing agents to learn from joint information while acting from local…
Multi-agent Reinforcement LearningDealing with Non-Stationarity in MARL via Trust-Region Decomposition
Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL). One of the reasons is the policy changes of agents during the learning process. Some existing works have discussed various co…
Multi-agent Reinforcement Learning