Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning

2019-06-09 · NeurIPS 2019 12 · Mahmoud Assran, Joshua Romoff, Nicolas Ballas, Joelle Pineau, Michael Rabbat

Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication topology, and exchange information through asynchronous gossip in order to take advantage of a large number of distributed simulators. We prove that GALA agents remain within an epsilon-ball of one-another during training when using loosely coupled asynchronous communication. By reducing the amount of synchronization between agents, GALA is more computationally efficient and scalable compared to A2C, its fully-synchronous counterpart. GALA also outperforms A2C, being more robust and sample efficient. We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.

📄 PDF Abstract BibTeX arXiv:1906.04585

Code (1)

facebookresearch/gala 공식 구현 pytorch

Tasks

Deep Reinforcement LearningGPUreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

GALA Most attention mechanisms learn where to focus using only weak supervisory signals from class labels, which inspired Linsley et al. to investigate how explicit human supervision…

A2C A2C, or Advantage Actor Critic, is a synchronous version of the A3C policy gradient method. As an alternative to the asynchronous…

Reinforcement Learning Architectures: SAC, TAC, and ESAC

2020-04-05 · Ala'eddin Masadeh, Zhengdao Wang, Ahmed E. Kamal

The trend is to implement intelligent agents capable of analyzing available information and utilize it efficiently. This work presents a number of reinforcement learning (RL) architectures; one of them is designed for in…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Elastic Gossip: Distributing Neural Network Training Using Gossip-like Protocols

2018-12-06 · Siddharth Pramod

Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resour…

Distributed Reinforcement Learning via Gossip

2013-10-28 · Adwaitvedant S. Mathkar, Vivek S. Borkar

We consider the classical TD(0) algorithm implemented on a network of agents wherein the agents also incorporate the updates received from neighboring agents using a gossip-like mechanism. The combined scheme is shown to…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Revisiting Gossip Protocols: A Vision for Emergent Coordination in Agentic Multi-Agent Systems

2025-08-03 · Mansura Habiba, Nafiul I. Khan arxiv

As agentic platforms scale, agents are evolving beyond static roles and fixed toolchains, creating a growing need for flexible, decentralized coordination. Today's structured communication protocols (e.g., direct agent-t…

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

2018-02-05 · ICML 2018 7 · Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan 외

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. …

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)