QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

2026-02-26 · Yuanjun Li, Bin Zhang, Hao Chen, Zhouyang Jiang, Dapeng Li, Zhiwei Xu arxiv

Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation leads to systematic Q-value overestimation. This issue is particularly severe in MARL due to the combinatorial explosion of the joint action space, which often results in unstable learning and suboptimal policies. To address this problem, we propose QSIM, a similarity weighted Q-learning framework that reconstructs the TD target using action similarity. Instead of using the greedy joint action directly, QSIM forms a similarity weighted expectation over a structured near-greedy joint action space. This formulation allows the target to integrate Q-values from diverse yet behaviorally related actions while assigning greater influence to those that are more similar to the greedy choice. By smoothing the target with structurally relevant alternatives, QSIM effectively mitigates overestimation and improves learning stability. Extensive experiments demonstrate that QSIM can be seamlessly integrated with various VD methods, consistently yielding superior performance and stability compared to the original algorithms. Furthermore, empirical analysis confirms that QSIM significantly mitigates the systematic value overestimation in MARL. Code is available at https://github.com/MaoMaoLYJ/pymarl-qsim.

📄 PDF Abstract BibTeX arXiv:2602.22786

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning

DRAS-CQSim: A Reinforcement Learning based Framework for HPC Cluster Scheduling

2021-05-16 · Yuping Fan, Zhiling Lan

For decades, system administrators have been striving to design and tune cluster scheduling policies to improve the performance of high performance computing (HPC) systems. However, the increasingly complex HPC systems c…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Scheduling

Dual Ensembled Multiagent Q-Learning with Hypernet Regularizer

2025-02-04 · Yaodong Yang, Guangyong Chen, Hongyao Tang, Furui Liu 외

Overestimation in single-agent reinforcement learning has been extensively studied. In contrast, overestimation in the multiagent setting has received comparatively little attention although it increases with the number …

Q-Learning SMAC SMAC+

Regularized Softmax Deep Multi-Agent Q-Learning

2021-12-01 · NeurIPS 2021 12 · Ling Pan, Tabish Rashid, Bei Peng, Longbo Huang 외

Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting. In thi…

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+3

Regularized Softmax Deep Multi-Agent $Q$-Learning

2021-03-22 · Ling Pan, Tabish Rashid, Bei Peng, Longbo Huang 외

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+3

Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics

2019-10-03 · Johannes Ackermann, Volker Gabler, Takayuki Osa, Masashi Sugiyama

Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn…

Multi-agent Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)