Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning

2023-06-02 · Dingyang Chen, Qi Zhang

Executing actions in a correlated manner is a common strategy for human coordination that often leads to better cooperation, which is also potentially beneficial for cooperative multi-agent reinforcement learning (MARL). However, the recent success of MARL relies heavily on the convenient paradigm of purely decentralized execution, where there is no action correlation among agents for scalability considerations. In this work, we introduce a Bayesian network to inaugurate correlations between agents' action selections in their joint policy. Theoretically, we establish a theoretical justification for why action dependencies are beneficial by deriving the multi-agent policy gradient formula under such a Bayesian network joint policy and proving its global convergence to Nash equilibria under tabular softmax policy parameterization in cooperative Markov games. Further, by equipping existing MARL algorithms with a recent method of differentiable directed acyclic graphs (DAGs), we develop practical algorithms to learn the context-aware Bayesian network policies in scenarios with partial observability and various difficulty. We also dynamically decrease the sparsity of the learned DAG throughout the training process, which leads to weakly or even purely independent policies for decentralized execution. Empirical results on a range of MARL benchmarks show the benefits of our approach.

📄 PDF Abstract BibTeX arXiv:2306.01920

Code (1)

dchen48/bnpg 공식 구현

Tasks

Multi-agent Reinforcement Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

2023-06-02 · Andrew Jesson, Chris Lu, Gunshi Gupta, Nicolas Beltran-Velez 외

This paper proposes a step toward approximate Bayesian inference in on-policy actor-critic deep reinforcement learning. It is implemented through three changes to the Asynchronous Advantage Actor-Critic (A3C) algorithm: …

Bayesian Inference continuous-control Continuous Control Deep Reinforcement Learning+4

Deep Exploration with PAC-Bayes

2024-02-05 · Bahareh Tasdighi, Manuel Haussmann, Nicklas Werge, Yi-Shan Wu 외

Reinforcement learning for continuous control under delayed rewards is an under-explored problem despite its significance in real life. Many complex skills build on intermediate ones as prerequisites. For instance, a hum…

continuous-control Continuous Control Decision Making reinforcement-learning+1

PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning

2024-05-10 · Jaejun Lee, Minsung Hwang, Joyce Jiyoung Whang

While a number of knowledge graph representation learning (KGRL) methods have been proposed over the past decade, very few theoretical analyses have been conducted on them. In this paper, we present the first PAC-Bayesia…

Decoder Generalization Bounds Graph Neural Network Graph Representation Learning+5

Bayesian Surrogates for Risk-Aware Pre-Assessment of Aging Bridge Portfolios

2025-09-29 · Sophia V. Kuhn, Rafael Bischof, Marius Weber, Antoine Binggeli 외 arxiv

Aging infrastructure portfolios pose a critical resource allocation challenge: deciding which structures require intervention and which can safely remain in service. Structural assessments must balance the trade-off betw…

Bayesian policy gradient and actor-critic algorithms

2026-04-30 · Mohammad Ghavamzadeh, Yaakov Engel, Michal Valko arxiv

Policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate. Conventional policy gradient methods use Monte-Carlo techniques to estimate th…

Reinforcement Learning