Structural Equivalence and Learning Dynamics in Delayed MARL

2026-05-05 · Jules Sintes, Ana Bušić, Jiamin Zhu arxiv

We formally establish the equivalence between Observation Delay (OD) and Action Delay (AD) in cooperative partially observable multi-agent systems using observation-action histories. We show that both systems generate identical admissible joint-policy sets, and their induced state-action-observation trajectories are identical in distribution, leading to identical optimal solutions in Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). This formally generalizes existing infinite-horizon single-agent results to any-horizon partially observable cooperative multi-agent problems with decentralized policy execution, and allows any mixed-delay configuration to be reduced to a pure OD system. We further prove that in Transition-Independent MDPs (TI-MDPs), the observation-action history reduces to a tractable minimal local augmented state. However, we show through numerical experiments that although the optimal solution spaces are structurally isomorphic, the practical learning dynamics are fundamentally different. First, using the minimal local augmented state, the equivalence no longer holds when transitions are not independent. Second, operational constraints and causal credit-assignment errors in Temporal Difference (TD) algorithms induce different learning behaviors across regimes. Finally, leveraging this structural equivalence to bypass these learning challenges, we demonstrate successful multi-agent zero-shot policy transfer from OD to AD, paving the way for unified, efficient solution methods in complex delayed systems.

📄 PDF Abstract BibTeX arXiv:2605.04345

Code (0)

등록된 구현이 없습니다.

Delayed homomorphic reinforcement learning for environments with delayed feedback

2026-04-04 · Jongsoo Lee, Jangwon Kim, Soohee Han arxiv

Reinforcement learning in real-world systems often involves delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches cause state-space explosion,…

Reinforcement Learning

PAC Reinforcement Learning Algorithm for General-Sum Markov Games

2020-09-05 · Ashkan Zehfroosh, Herbert G. Tanner

This paper presents a theoretical framework for probably approximately correct (PAC) multi-agent reinforcement learning (MARL) algorithms for Markov games. The paper offers an extension to the well-known Nash Q-learning …

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction

2018-09-25 · Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen 외

Multiagent reinforcement learning (MARL) is commonly considered to suffer from non-stationary environments and exponentially increasing policy space. It would be even more challenging when rewards are sparse and delayed …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

2022-10-31 · Jennifer She, Jayesh K. Gupta, Mykel J. Kochenderfer

Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not on…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering

2026-05-25 · Maxim Mednikov, Oren Gal arxiv

Real-world multi-agent reinforcement learning (MARL) systems must often operate under stale observations, stochastic communication delays, and intermittent packet loss. Policies trained under idealized synchronous condit…

Multi-agent Reinforcement Learning