Challenges in Credit Assignment for Multi-Agent Reinforcement Learning in Open Agent Systems

2025-10-31 · Alireza Saleh Abadi, Leen-Kiat Soh arxiv

In the rapidly evolving field of multi-agent reinforcement learning (MARL), understanding the dynamics of open systems is crucial. Openness in MARL refers to the dynam-ic nature of agent populations, tasks, and agent types with-in a system. Specifically, there are three types of openness as reported in (Eck et al. 2023) [2]: agent openness, where agents can enter or leave the system at any time; task openness, where new tasks emerge, and existing ones evolve or disappear; and type openness, where the capabil-ities and behaviors of agents change over time. This report provides a conceptual and empirical review, focusing on the interplay between openness and the credit assignment problem (CAP). CAP involves determining the contribution of individual agents to the overall system performance, a task that becomes increasingly complex in open environ-ments. Traditional credit assignment (CA) methods often assume static agent populations, fixed and pre-defined tasks, and stationary types, making them inadequate for open systems. We first conduct a conceptual analysis, in-troducing new sub-categories of openness to detail how events like agent turnover or task cancellation break the assumptions of environmental stationarity and fixed team composition that underpin existing CAP methods. We then present an empirical study using representative temporal and structural algorithms in an open environment. The results demonstrate that openness directly causes credit misattribution, evidenced by unstable loss functions and significant performance degradation.

📄 PDF Abstract BibTeX arXiv:2510.27659

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning

Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

2025-05-13 · Shuai Han, Mehdi Dastani, Shihan Wang

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods…

Efficient Exploration Multi-agent Reinforcement Learning

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

2021-06-01 · Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu 외

Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challe…

counterfactual Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3

MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment

2023-12-06 · Ziyan Wang, Yali Du, Yudi Zhang, Meng Fang 외

Offline Multi-agent Reinforcement Learning (MARL) is valuable in scenarios where online interaction is impractical or risky. While independent learning in MARL offers flexibility and scalability, accurately assigning cre…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Shapley Value Based Multi-Agent Reinforcement Learning: Theory, Method and Its Application to Energy Network

2024-02-23 · Jianhong Wang

Multi-agent reinforcement learning is an area of rapid advancement in artificial intelligence and machine learning. One of the important questions to be answered is how to conduct credit assignment in a multi-agent syste…

Learning Theory Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

2025-08-09 · Xutong Zhao, Yaqi Xie arxiv

Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents to achieve a common goal. A key challenge in MARL is credit assignment, which involves assessing each agent's contribution to the s…