Divergence-Regularized Multi-Agent Actor-Critic

2021-10-01 · Kefan Su, Zongqing Lu

Entropy regularization is a popular method in reinforcement learning (RL). Although it has many advantages, it alters the RL objective of the original Markov Decision Process (MDP). Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC). Theoretically, we derive the update rule of DMAC which is naturally off-policy and guarantees monotonic policy improvement and convergence in both the original MDP and divergence-regularized MDP. We also give a bound of the discrepancy between the converged policy and optimal policy in the original MDP. DMAC is a flexible framework and can be combined with many existing MARL algorithms. Empirically, we evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and show that DMAC substantially improves the performance of existing MARL algorithms.

📄 PDF Abstract BibTeX arXiv:2110.00304

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Starcraft

ARAC: Adaptive Regularized Multi-Agent Soft Actor-Critic in Graph-Structured Adversarial Games

2025-11-11 · Ruochuan Shi, Runyu Lu, Yuanheng Zhu, Dongbin Zhao arxiv

In graph-structured multi-agent reinforcement learning (MARL) adversarial tasks such as pursuit and confrontation, agents must coordinate under highly dynamic interactions, where sparse rewards hinder efficient policy le…

Multi-agent Reinforcement Learning Graph Neural Network

GRAC: Self-Guided and Self-Regularized Actor-Critic

2020-09-18 · Lin Shao, Yifan You, Mengyuan Yan, Qingyun Sun 외

Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the…

Decision Making Deep Reinforcement Learning OpenAI Gym reinforcement-learning+2

Mirror Descent Actor Critic via Bounded Advantage Learning

2025-02-06 · Ryo Iwaki

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates…

Reinforcement Learning (RL)

BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning

2021-10-02 · Chi Zhang, Sanmukh Rao Kuppannagari, Viktor K Prasanna

Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Symmetric Behavior Regularized Policy Optimization

2025-08-06 · Lingwei Zhu, Haseeb Shah, Zheng Chen, Yukie Nagai 외 arxiv

Behavior Regularized Policy Optimization (BRPO) leverages asymmetric (divergence) regularization to mitigate the distribution shift in offline Reinforcement Learning. This paper is the first to study the open question of…

Reinforcement Learning Offline RL