MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer

2022-06-20 · Jeewon Jeon, Woojun Kim, Whiyoung Jung, Youngchul Sung

In this paper, we consider cooperative multi-agent reinforcement learning (MARL) with sparse reward. To tackle this problem, we propose a novel method named MASER: MARL with subgoals generated from experience replay buffer. Under the widely-used assumption of centralized training with decentralized execution and consistent Q-value decomposition for MARL, MASER automatically generates proper subgoals for multiple agents from the experience replay buffer by considering both individual Q-value and total Q-value. Then, MASER designs individual intrinsic reward for each agent based on actionable representation relevant to Q-learning so that the agents reach their subgoals while maximizing the joint action value. Numerical results show that MASER significantly outperforms StarCraft II micromanagement benchmark compared to other state-of-the-art MARL algorithms.

📄 PDF Abstract BibTeX arXiv:2206.10607

Code (1)

jiwonjeon9603/maser 공식 구현 pytorch

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning (RL)StarcraftStarcraft II

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning

2019-01-24 · Sanjeevan Ahilan, Peter Dayan

We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, w…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction

2025-02-08 · Shengbin Yue, Ting Huang, Zheng Jia, Siyuan Wang 외

Large Language Models (LLMs) have significantly advanced legal intelligence, but the scarcity of scenario data impedes the progress toward interactive legal scenarios. This paper introduces a Multi-agent Legal Simulation…

Words as Beacons: Guiding RL Agents with High-Level Language Prompts

2024-10-11 · Unai Ruiz-Gonzalez, Alain Andres, Pedro G. Bascoy, Javier Del Ser

Sparse reward environments in reinforcement learning (RL) pose significant challenges for exploration, often leading to inefficient or incomplete learning processes. To tackle this issue, this work proposes a teacher-stu…

Reinforcement Learning (RL)

Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning

2019-05-13 · Libo Xing

Hierarchical Reinforcement Learning (HRL) exploits temporally extended actions, or options, to make decisions from a higher-dimensional perspective to alleviate the sparse reward problem, one of the most challenging prob…

Hierarchical Reinforcement Learning Montezuma's Revenge reinforcement-learning Reinforcement Learning+1

Hierarchical Reinforcement Learning with Timed Subgoals

2021-12-06 · NeurIPS 2021 12 · Nico Gürtler, Dieter Büchler, Georg Martius

Hierarchical reinforcement learning (HRL) holds great potential for sample-efficient learning on challenging long-horizon tasks. In particular, letting a higher level assign subgoals to a lower level has been shown to en…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)