MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment

2023-12-06 · Ziyan Wang, Yali Du, Yudi Zhang, Meng Fang, Biwei Huang

Offline Multi-agent Reinforcement Learning (MARL) is valuable in scenarios where online interaction is impractical or risky. While independent learning in MARL offers flexibility and scalability, accurately assigning credit to individual agents in offline settings poses challenges because interactions with an environment are prohibited. In this paper, we propose a new framework, namely Multi-Agent Causal Credit Assignment (MACCA), to address credit assignment in the offline MARL setting. Our approach, MACCA, characterizing the generative process as a Dynamic Bayesian Network, captures relationships between environmental variables, states, actions, and rewards. Estimating this model on offline data, MACCA can learn each agent's contribution by analyzing the causal relationship of their individual rewards, ensuring accurate and interpretable credit assignment. Additionally, the modularity of our approach allows it to seamlessly integrate with various offline MARL methods. Theoretically, we proved that under the setting of the offline dataset, the underlying causal structure and the function for generating the individual rewards of agents are identifiable, which laid the foundation for the correctness of our modeling. In our experiments, we demonstrate that MACCA not only outperforms state-of-the-art methods but also enhances performance when integrated with other backbones.

📄 PDF Abstract BibTeX arXiv:2312.03644

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?

2023-12-28 · Gunshi Gupta, Tim G. J. Rudner, Rowan Thomas McAllister, Adrien Gaidon 외

Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during training if most of the training data conta…

reinforcement-learning Reinforcement Learning

Causal Reinforcement Learning using Observational and Interventional Data

2021-06-28 · Maxime Gasse, Damien Grasset, Guillaume Gaudron, Pierre-Yves Oudeyer

Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences t…

Causal Inference Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Why Online Reinforcement Learning is Causal

2024-03-07 · Oliver Schulte, Pascal Poupart

Reinforcement learning (RL) and causal modelling naturally complement each other. The goal of causal modelling is to predict the effects of interventions in an environment, while the goal of reinforcement learning is to …

counterfactual Offline RL reinforcement-learning Reinforcement Learning+1

Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning

2023-04-17 · Siyu Wang, Xiaocong Chen, Dietmar Jannach, Lina Yao

Reinforcement learning-based recommender systems have recently gained popularity. However, the design of the reward function, on which the agent relies to optimize its recommendation policy, is often not straightforward.…

Recommendation Systems reinforcement-learning Reinforcement Learning

Learning Implicit Causal World Models from Multi-Agent Demonstrations

2026-07-28 · Jasorsi Ghosh arxiv

In model-based reinforcement learning, world models exist as internal simulators, but their training often conflates statistical correlations with causal mechanisms. This problem is exacerbated in multi-agent systems whe…

Reinforcement Learning