Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

2024-12-10 · Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, Xuetao Ding

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.

📄 PDF Abstract BibTeX arXiv:2412.07639

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning

Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration

2024-10-25 · Hai Zhong, Xun Wang, Zhuoran Li, Longbo Huang

Offline-to-Online Reinforcement Learning has emerged as a powerful paradigm, leveraging offline data for initialization and online fine-tuning to enhance both sample efficiency and performance. However, most existing res…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3

Offline Pre-trained Multi-Agent Decision Transformer

2021-09-29 · Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le 외

Offline reinforcement learning leverages static datasets to learn optimal policies with no necessity to access the environment. This is desirable for multi-agent systems due to the expensiveness of agents' online interac…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Accelerating exploration and representation learning with offline pre-training

2023-03-31 · Bogdan Mazoure, Jake Bruce, Doina Precup, Rob Fergus 외

Sequential decision-making agents struggle with long horizon tasks, since solving them requires multi-step reasoning. Most reinforcement learning (RL) algorithms address this challenge by improved credit assignment, intr…

Decision Making NetHack Reinforcement Learning (RL)Representation Learning+1

Reinforcement Learning: An Overview

2024-12-06 · Kevin Murphy

This manuscript gives a big-picture, up-to-date overview of the field of (deep) reinforcement learning and sequential decision making, covering value-based methods, policy-based methods, model-based methods, multi-agent …

Decision Making Deep Reinforcement Learning Offline RL Policy Gradient Methods+3

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

2023-07-21 · NeurIPS 2023 11 · Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success…

Management MuJoCo Multi-agent Reinforcement Learning Reinforcement Learning (RL)+2