PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning

2022-06-22 · Hanhan Zhou, Tian Lan, Vaneet Aggarwal

Multi-agent reinforcement learning (MARL) has witnessed significant progress with the development of value function factorization methods. It allows optimizing a joint action-value function through the maximization of factorized per-agent utilities due to monotonicity. In this paper, we show that in partially observable MARL problems, an agent's ordering over its own actions could impose concurrent constraints (across different states) on the representable function class, causing significant estimation error during training. We tackle this limitation and propose PAC, a new framework leveraging Assistive information generated from Counterfactual Predictions of optimal joint action selection, which enable explicit assistance to value function factorization through a novel counterfactual loss. A variational inference-based information encoding method is developed to collect and encode the counterfactual predictions from an estimated baseline. To enable decentralized execution, we also derive factorized per-agent policies inspired by a maximum-entropy MARL framework. We evaluate the proposed PAC on multi-agent predator-prey and a set of StarCraft II micromanagement tasks. Empirical results demonstrate improved results of PAC over state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms on all benchmarks.

📄 PDF Abstract BibTeX arXiv:2206.11420

Code (1)

hanhananderson/pac-marl 공식 구현 pytorch

Tasks

counterfactualMulti-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)StarcraftStarcraft IIVariational Inference

Bayesian Hybrid Matrix Factorisation for Data Integration

2017-04-17 · Thomas Brouwer, Pietro Lió

We introduce a novel Bayesian hybrid matrix factorisation model (HMF) for data integration, based on combining multiple matrix factorisation methods, that can be used for in- and out-of-matrix prediction of missing value…

Data Integration Missing Values

Prior and Likelihood Choices for Bayesian Matrix Factorisation on Small Datasets

2017-12-01 · Thomas Brouwer, Pietro Lio'

In this paper, we study the effects of different prior and likelihood choices for Bayesian matrix factorisation, focusing on small datasets. These choices can greatly influence the predictive performance of the methods. …

Model Selection

SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning

2021-05-31 · Jianhong Wang, Yuan Zhang, Yunjie Gu, Tae-Kyun Kim

Value factorisation is a useful technique for multi-agent reinforcement learning (MARL) in global reward game, however its underlying mechanism is not yet fully understood. This paper studies a theoretical framework for …

Fairness Multi-agent Reinforcement Learning Q-Learning SMAC+1

QGNN: Value Function Factorisation with Graph Neural Networks

2022-05-25 · Ryan Kortvelesy, Amanda Prorok

In multi-agent reinforcement learning, the use of a global objective is a powerful tool for incentivising cooperation. Unfortunately, it is not sample-efficient to train individual agents with a global reward, because it…

Graph Neural Network Multi-agent Reinforcement Learning Starcraft

Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations

2025-09-27 · Sihao Ding, Santosh Vasa, Aditi Ramadwar arxiv

Vision-Language Models (VLMs) often produce fluent Natural Language Explanations (NLEs) that sound convincing but may not reflect the causal factors driving predictions. This mismatch of plausibility and faithfulness pos…