Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning

2025-06-23 · Azad Deihim, Eduardo Alonso, Dimitra Apostolopoulou

We present the Multi-Agent Transformer World Model (MATWM), a novel transformer-based world model designed for multi-agent reinforcement learning in both vector- and image-based environments. MATWM combines a decentralized imagination framework with a semi-centralized critic and a teammate prediction module, enabling agents to model and anticipate the behavior of others under partial observability. To address non-stationarity, we incorporate a prioritized replay mechanism that trains the world model on recent experiences, allowing it to adapt to agents' evolving policies. We evaluated MATWM on a broad suite of benchmarks, including the StarCraft Multi-Agent Challenge, PettingZoo, and MeltingPot. MATWM achieves state-of-the-art performance, outperforming both model-free and prior world model approaches, while demonstrating strong sample efficiency, achieving near-optimal performance in as few as 50K environment interactions. Ablation studies confirm the impact of each component, with substantial gains in coordination-heavy tasks.

📄 PDF Abstract BibTeX arXiv:2506.18537

Code (1)

azaddeihim/matwm 공식 구현

Tasks

Multi-agent Reinforcement LearningStarcraft

Methods 이 논문이 사용한 방법론

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models

2024-06-22 · Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan 외

Learning a world model for model-free Reinforcement Learning (RL) agents can significantly improve the sample efficiency by learning policies in imagination. However, building a world model for Multi-Agent RL (MARL) can …

Reinforcement Learning (RL)SMAC SMAC+Starcraft

Transformers are Sample-Efficient World Models

2022-09-01 · Vincent Micheli, Eloi Alonso, François Fleuret

Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with…

Atari Games 100k Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

TransDreamer: Reinforcement Learning with Transformer World Models

2022-02-19 · Chang Chen, Yi-Fu Wu, Jaesik Yoon, Sungjin Ahn

The Dreamer agent provides various benefits of Model-Based Reinforcement Learning (MBRL) such as sample efficiency, reusable knowledge, and safe planning. However, its world model and policy networks inherit the limitati…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Object-Centric World Models for Causality-Aware Reinforcement Learning

2025-11-18 · Yosuke Nishimoto, Takashi Matsubara arxiv

World models have been developed to support sample-efficient deep reinforcement learning agents. However, it remains challenging for world models to accurately replicate environments that are high-dimensional, non-statio…

Reinforcement Learning

Offline Pre-trained Multi-Agent Decision Transformer

2021-09-29 · Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le 외

Offline reinforcement learning leverages static datasets to learn optimal policies with no necessity to access the environment. This is desirable for multi-agent systems due to the expensiveness of agents' online interac…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1