Associative Memory Based Experience Replay for Deep Reinforcement Learning

2022-07-16 · Mengyuan Li, Arman Kazemi, Ann Franchesca Laguna, X. Sharon Hu

Experience replay is an essential component in deep reinforcement learning (DRL), which stores the experiences and generates experiences for the agent to learn in real time. Recently, prioritized experience replay (PER) has been proven to be powerful and widely deployed in DRL agents. However, implementing PER on traditional CPU or GPU architectures incurs significant latency overhead due to its frequent and irregular memory accesses. This paper proposes a hardware-software co-design approach to design an associative memory (AM) based PER, AMPER, with an AM-friendly priority sampling operation. AMPER replaces the widely-used time-costly tree-traversal-based priority sampling in PER while preserving the learning performance. Further, we design an in-memory computing hardware architecture based on AM to support AMPER by leveraging parallel in-memory search operations. AMPER shows comparable learning performance while achieving 55x to 270x latency improvement when running on the proposed hardware compared to the state-of-the-art PER running on GPU.

📄 PDF Abstract BibTeX arXiv:2207.07791

Code (0)

등록된 구현이 없습니다.

Tasks

CPUDeep Reinforcement LearningGPUreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

AM 설명 없음

Prioritized Experience Replay Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Saliency-Guided Hidden Associative Replay for Continual Learning

2023-10-06 · Guangji Bai, Qilong Zhao, Xiaoyang Jiang, Yifei Zhang 외

Continual Learning is a burgeoning domain in next-generation AI, focusing on training neural networks over a sequence of tasks akin to human learning. While CL provides an edge over traditional supervised learning, its c…

Continual Learning Retrieval

Episodic Reinforcement Learning with Associative Memory

2020-05-01 · ICLR 2020 1 · Guangxiang Zhu*, Zichuan Lin*, Guangwen Yang, Chongjie Zhang

Sample efficiency has been one of the major challenges for deep reinforcement learning. Non-parametric episodic control has been proposed to speed up parametric reinforcement learning by rapidly latching on previously su…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Prioritized Experience Replay

2015-11-18 · Tom Schaul, John Quan, Ioannis Antonoglou, David Silver

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply r…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Deep In-GPU Experience Replay

2018-01-09 · Ben Parr

Experience replay allows a reinforcement learning agent to train on samples from a large amount of the most recent experiences. A simple in-RAM experience replay stores these most recent experiences in a list in RAM, and…

CPU GPU Reinforcement Learning

The Effects of Memory Replay in Reinforcement Learning

2017-10-18 · Ruishan Liu, James Zou

Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despit…

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1