Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning

2026-05-12 · Yue Deng, Zirui Wang, Yin Zhang arxiv

TD($λ$) in value-based MARL algorithms or the Temporal Difference critic learning in Actor-Critic-based (AC-based) algorithms synergistically integrate elements from Monte-Carlo simulation and Q function bootstrapping via dynamic programming, which effectively addresses the inherent bias-variance trade-off in value estimation. Based on that, some recent works link the adaptive $λ$ value to the policy distribution in the single-agent reinforcement learning area. However, because of the large joint action space from multiple number of agents, and the limited transition data in Multi-agent Reinforcement Learning, the policy distribution is infeasible to be calculated statistically. To solve the policy distribution calculation problem in MARL settings, we employ a parametric likelihood-free density ratio estimator with two replay buffers instead of calculating statistically. The two replay buffers of different sizes store the historical trajectories that represent the data distribution of the past and current policies correspondingly. Based on the estimator, we assign Adaptive TD($λ$), \textbf{ATD($λ$)}, values to state-action pairs based on their likelihood under the stationary distribution of the current policy. We apply the proposed method on two competitive baseline methods, QMIX for value-based algorithms, and MAPPO for AC-based algorithms, over SMAC benchmarks and Gfootball academy scenarios, and demonstrate consistently competitive or superior performance compared to other baseline approaches with static $λ$ values.

📄 PDF Abstract BibTeX arXiv:2605.11880

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning

SMIX($λ$): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

2019-11-11 · Xinghu Yao, Chao Wen, Yuhui Wang, Xiaoyang Tan

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+3

Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork

2025-06-12 · Jianhong Wang, Yang Li, Samuel Kaski, Jonathan Lawry

Open multi-agent systems are increasingly important in modeling real-world applications, such as smart grids, swarm robotics, etc. In this paper, we aim to investigate a recently proposed problem for open multi-agent sys…

Reinforcement Learning (RL)

Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning

2024-12-20 · Lunjun Liu, Weilai Jiang, Yaonan Wang

In multi-agent reinforcement learning (MARL), the centralized training with decentralized execution (CTDE) framework has gained widespread adoption due to its strong performance. However, the further development of CTDE …

Decision Making Multi-agent Reinforcement Learning

AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning

2024-12-20 · Guangchong Zhou, Zeren Zhang, Guoliang Fan

Exploration in cooperative multi-agent reinforcement learning (MARL) remains challenging for value-based agents due to the absence of an explicit policy. Existing approaches include individual exploration based on uncert…

Diversity Multi-agent Reinforcement Learning

Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

2021-06-01 · Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li 외

Extending transfer learning to cooperative multi-agent reinforcement learning (MARL) has recently received much attention. In contrast to the single-agent setting, the coordination indispensable in cooperative MARL const…

Management Multi-agent Reinforcement Learning Starcraft Starcraft II+1