Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning

2024-03-02 · Hyungho Na, Yunkyeong Seo, Il-Chul Moon

In cooperative multi-agent reinforcement learning (MARL), agents aim to achieve a common goal, such as defeating enemies or scoring a goal. Existing MARL algorithms are effective but still require significant learning time and often get trapped in local optima by complex tasks, subsequently failing to discover a goal-reaching policy. To address this, we introduce Efficient episodic Memory Utilization (EMU) for MARL, with two primary objectives: (a) accelerating reinforcement learning by leveraging semantically coherent memory from an episodic buffer and (b) selectively promoting desirable transitions to prevent local convergence. To achieve (a), EMU incorporates a trainable encoder/decoder structure alongside MARL, creating coherent memory embeddings that facilitate exploratory memory recall. To achieve (b), EMU introduces a novel reward structure called episodic incentive based on the desirability of states. This reward improves the TD target in Q-learning and acts as an additional incentive for desirable transitions. We provide theoretical support for the proposed incentive and demonstrate the effectiveness of EMU compared to conventional episodic control. The proposed method is evaluated in StarCraft II and Google Research Football, and empirical results indicate further performance improvement over state-of-the-art methods.

📄 PDF Abstract BibTeX arXiv:2403.01112

Code (1)

hyunghona/emu 공식 구현 pytorch

Tasks

Decoder Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement LearningStarcraftStarcraft II

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

2026-06-03 · Zicheng Zhao, Yu Lan, Chengzhengxu Li, Zhaohan Zhang 외 arxiv

Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks. While episodic memory mechanisms mitigate these issues by reusing high-return trajectories…

Multi-agent Reinforcement Learning Contrastive Learning

Spatially-Aware Transformer for Embodied Agents

2024-02-23 · Junmo Cho, Jaesik Yoon, Sungjin Ahn

Episodic memory plays a crucial role in various cognitive processes, such as the ability to mentally recall past events. While cognitive science emphasizes the significance of spatial context in the formation and retriev…

reinforcement-learning Reinforcement Learning

Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration

2021-11-22 · NeurIPS 2021 12 · Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He 외

Efficient exploration in deep cooperative multi-agent reinforcement learning (MARL) still remains challenging in complex coordination problems. In this paper, we introduce a novel Episodic Multi-agent reinforcement learn…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3

Now I Remember! Episodic Memory For Reinforcement Learning

2018-01-01 · ICLR 2018 1 · Ricky Loynd, Matthew Hausknecht, Lihong Li, Li Deng

Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car. Endowing reinforcement learning agents with episodic…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Two-Memory Reinforcement Learning

2023-04-20 · Zhao Yang, Thomas. M. Moerland, Mike Preuss, Aske Plaat

While deep reinforcement learning has shown important empirical success, it tends to learn relatively slow due to slow propagation of rewards information and slow update of parametric neural networks. Non-parametric epis…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning+1