Fully Asynchronous Policy Evaluation in Distributed Reinforcement Learning over Networks

2020-03-01 · Xingyu Sha, Jia-Qi Zhang, Keyou You, Kaiqing Zhang, Tamer Başar

This paper proposes a \emph{fully asynchronous} scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks. Without waiting for any other node of the network, each node can locally update its value function at any time by using (possibly delayed) information from its neighbors. This is in sharp contrast to the gossip-based scheme where a pair of nodes concurrently update. Though the fully asynchronous setting involves a difficult multi-timescale decision problem, we design a novel stochastic average gradient (SAG) based distributed algorithm and develop a push-pull augmented graph approach to prove its exact convergence at a linear rate of $\mathcal{O}(c^k)$ where $c\in(0,1)$ and $k$ increases by one no matter on which node updates. Finally, numerical experiments validate that our method speeds up linearly with respect to the number of nodes, and is robust to straggler nodes.

📄 PDF Abstract BibTeX arXiv:2003.00433

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

A3C A3C, Asynchronous Advantage Actor Critic, is a policy gradient algorithm in reinforcement learning that maintains a policy $\pi\left(a\_{t}\mid{s}\_{t}; \theta\right)$ and…

Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning

2025-09-29 · Alexander Tyurin, Andrei Spiridonov, Varvara Rudenko arxiv

We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achi…

Reinforcement Learning

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainin

2025-05-29 · Bo Wu, Sid Wang, Yunhao Tang, Jia Ding 외

Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, because of the high demands on latency and memory, it is parti…

GPU Reinforcement Learning (RL)

Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search

2016-10-03 · Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar 외

In principle, reinforcement learning and policy search methods can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, tr…

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning

2025-05-12 · Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong 외

We introduce INTELLECT-2, the first globally distributed reinforcement learning (RL) training run of a 32 billion parameter language model. Unlike traditional centralized training efforts, INTELLECT-2 trains a reasoning …