Accelerating Distributed Deep Reinforcement Learning by In-Network Experience Sampling

2021-10-26 · Masaki Furukawa, Hiroki Matsutani

A computing cluster that interconnects multiple compute nodes is used to accelerate distributed reinforcement learning based on DQN (Deep Q-Network). In distributed reinforcement learning, Actor nodes acquire experiences by interacting with a given environment and a Learner node optimizes their DQN model. Since data transfer between Actor and Learner nodes increases depending on the number of Actor nodes and their experience size, communication overhead between them is one of major performance bottlenecks. In this paper, their communication is accelerated by DPDK-based network optimizations, and DPDK-based low-latency experience replay memory server is deployed between Actor and Learner nodes interconnected with a 40GbE (40Gbit Ethernet) network. Evaluation results show that, as a network optimization technique, kernel bypassing by DPDK reduces network access latencies to a shared memory server by 32.7% to 58.9%. As another network optimization technique, an in-network experience replay memory server between Actor and Learner nodes reduces access latencies to the experience replay memory by 11.7% to 28.1% and communication latencies for prioritized experience sampling by 21.9% to 29.1%.

📄 PDF Abstract BibTeX arXiv:2110.13506

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Quantum deep Q learning with distributed prioritized experience replay

2023-04-19 · Samuel Yen-Chi Chen

This paper introduces the QDQN-DPER framework to enhance the efficiency of quantum reinforcement learning (QRL) in solving sequential decision tasks. The framework incorporates prioritized experience replay and asynchron…

Q-Learning reinforcement-learning Reinforcement Learning

M$^2$DQN: A Robust Method for Accelerating Deep Q-learning Network

2022-09-16 · Zhe Zhang, Yukun Zou, Junjie Lai, Qing Xu

Deep Q-learning Network (DQN) is a successful way which combines reinforcement learning with deep neural networks and leads to a widespread application of reinforcement learning. One challenging problem when applying DQN…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach

2020-02-06 · Zeyue Xue, Shuang Luo, Chao Wu, Pan Zhou 외

Peer-to-peer knowledge transfer in distributed environments has emerged as a promising method since it could accelerate learning and improve team-wide performance without relying on pre-trained teachers in deep reinforce…

Deep Reinforcement Learning Reinforcement Learning Transfer Learning

Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffer

2026-01-04 · Milind Nakul, Tianjiao Li, Ashwin Pananjady arxiv

We introduce the Multiscale Experience Replay (MER) algorithm for solving a class of stochastic variational inequalities (VIs) in settings where samples are generated from a Markov chain and we have access to a memory bu…

Accelerating Large Language Model Decoding with Speculative Sampling

2023-02-02 · Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau 외

We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of pa…

Language Modeling Language Modelling Large Language Model model