Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation

2025-05-06 · Songchen Fu, Siang Chen, Shaojing Zhao, Letian Bai, Ta Li, Yonghong Yan

In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation often consists of multiple components from other agents or dynamic entities in the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://anonymous.4open.science/r/RDC-pymarl-4512/.

📄 PDF Abstract BibTeX arXiv:2505.03586

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning SMAC SMAC+

Faster Deep Reinforcement Learning with Slower Online Network

2021-12-10 · Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim 외

Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay. Using two separate networks enables t…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Wideband Beamforming with Rainbow Beam Training using Reconfigurable True-Time-Delay Arrays for Millimeter-Wave Wireless

2021-11-30 · Chung-Ching Lin, Veljko Boljanovic, Han Yan, Erfan Ghaderi 외

The decadal research in integrated true-time-delay arrays have seen organic growth enabling realization of wideband beamformers for large arrays with wide aperture widths. This article introduces highly reconfigurable de…

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

2024-11-06 · Tyler Clark, Mark Towers, Christine Evers, Jonathon Hare

Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a n…

Computational Efficiency Deep Reinforcement Learning Reinforcement Learning (RL)

To Combine or Not To Combine? A Rainbow Deep Reinforcement Learning Agent for Dialog Policies

2019-09-01 · WS 2019 9 · Dirk V{\"a}th, Ngoc Thang Vu

In this paper, we explore state-of-the-art deep reinforcement learning methods for dialog policy training such as prioritized experience replay, double deep Q-Networks, dueling network architectures and distributional le…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learn to Interpret Atari Agents

2018-12-29 · Zhao Yang, Song Bai, Li Zhang, Philip H. S. Torr

Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks. However, the direct mapping from states to actions makes it hard to interpret the rationale behind the decision-making of the age…

Decision Making Deep Reinforcement Learning Reinforcement Learning