Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

2023-07-21 · NeurIPS 2023 11 · Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.

📄 PDF Abstract BibTeX arXiv:2307.11620

Code (1)

zhengyinan-air/omiga 공식 구현 pytorch

Tasks

ManagementMuJoCoMulti-agent Reinforcement Learning Reinforcement Learning (RL)StarcraftStarcraft II

Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization

2023-06-15 · Xiangsen Wang, Xianyuan Zhan

Offline reinforcement learning (RL) that learns policies from offline datasets without environment interaction has received considerable attention in recent years. Compared with the rich literature in the single-agent ca…

Management Multi-agent Reinforcement Learning Offline RL reinforcement-learning+4

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

2021-11-22 · Ling Pan, Longbo Huang, Tengyu Ma, Huazhe Xu

Conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, as many real-world scenarios involve interaction among multiple agents, it i…

Continuous Control Multi-agent Reinforcement Learning Offline RL reinforcement-learning+1

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

2021-06-07 · NeurIPS 2021 12 · Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng 외

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counte…

Multi-agent Reinforcement Learning Offline RL Q-Learning reinforcement-learning+3

Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning

2026-01-12 · Sijia Li, Xinran Li, Shibo Chen, Jun Zhang arxiv

Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within …

Multi-agent Reinforcement Learning

Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning

2021-09-29 · Jiechuan Jiang, Zongqing Lu

Offline reinforcement learning could learn effective policies from a fixed dataset, which is promising in real-world applications. However, in offline decentralized multi-agent reinforcement learning, due to the discrepa…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)