Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization

2023-06-15 · Xiangsen Wang, Xianyuan Zhan

Offline reinforcement learning (RL) that learns policies from offline datasets without environment interaction has received considerable attention in recent years. Compared with the rich literature in the single-agent case, offline multi-agent RL is still a relatively underexplored area. Most existing methods directly apply offline RL ingredients in the multi-agent setting without fully leveraging the decomposable problem structure, leading to less satisfactory performance in complex tasks. We present OMAC, a new offline multi-agent RL algorithm with coupled value factorization. OMAC adopts a coupled value factorization scheme that decomposes the global value function into local and shared components, and also maintains the credit assignment consistency between the state-value and Q-value functions. Moreover, OMAC performs in-sample learning on the decomposed local state-value functions, which implicitly conducts max-Q operation at the local level while avoiding distributional shift caused by evaluating out-of-distribution actions. Based on the comprehensive evaluations of the offline multi-agent StarCraft II micro-management tasks, we demonstrate the superior performance of OMAC over the state-of-the-art offline multi-agent RL methods.

📄 PDF Abstract BibTeX arXiv:2306.08900

Code (0)

등록된 구현이 없습니다.

Tasks

Management Multi-agent Reinforcement Learning Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)StarcraftStarcraft II

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

2023-07-21 · NeurIPS 2023 11 · Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success…

Management MuJoCo Multi-agent Reinforcement Learning Reinforcement Learning (RL)+2

Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration

2024-10-25 · Hai Zhong, Xun Wang, Zhuoran Li, Longbo Huang

Offline-to-Online Reinforcement Learning has emerged as a powerful paradigm, leveraging offline data for initialization and online fine-tuning to enhance both sample efficiency and performance. However, most existing res…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3

Offline Decentralized Multi-Agent Reinforcement Learning

2021-08-04 · Jiechuan Jiang, Zongqing Lu

In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. Howe…

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

A Recipe for Stable Offline Multi-agent Reinforcement Learning

2026-03-09 · Dongsu Lee, Daehee Lee, Amy Zhang arxiv

Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One …

Multi-agent Reinforcement Learning

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

2024-12-10 · Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu 외

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large join…