홈 › Papers

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

2017-02-28 · ICML 2017 8 · Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H. S. Torr, Pushmeet Kohli, Shimon Whiteson

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.

📄 PDF Abstract BibTeX arXiv:1702.08887

Code (5)

MUmarJaved/MultiAgent-Distributed-Reinforcement-Learning tf

anonymous1234517/code tf

cts198859/deeprl_dist tf

cts198859/deeprl_network tf

dongchen06/macacc pytorch

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Starcraft

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Learning offline: memory replay in biological and artificial reinforcement learning

2021-09-21 · Emma L. Roscow, Raymond Chua, Rui Ponte Costa, Matt W. Jones 외

Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in mac…

Continual Learning Decision Making reinforcement-learning Reinforcement Learning+1

Experience Replay Optimization

2019-06-19 · Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, Xia Hu

Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences …

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization

2023-02-21 · Yongsheng Mei, Hanhan Zhou, Tian Lan, Guru Venkataramani 외

Experience replay is crucial for off-policy reinforcement learning (RL) methods. By remembering and reusing the experiences from past different policies, experience replay significantly improves the training efficiency a…

Decision Making Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+2

Prioritized Experience Replay

2015-11-18 · Tom Schaul, John Quan, Ioannis Antonoglou, David Silver

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply r…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer

2022-06-20 · Jeewon Jeon, Woojun Kim, Whiyoung Jung, Youngchul Sung

In this paper, we consider cooperative multi-agent reinforcement learning (MARL) with sparse reward. To tackle this problem, we propose a novel method named MASER: MARL with subgoals generated from experience replay buff…

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning (RL)+2