Experience Replay Optimization

2019-06-19 · Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, Xia Hu

Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consider learning a replay policy to optimize the cumulative reward. Replay learning is challenging because the replay memory is noisy and large, and the cumulative reward is unstable. To address these issues, we propose a novel experience replay optimization (ERO) framework which alternately updates two policies: the agent policy, and the replay policy. The agent is updated to maximize the cumulative reward based on the replayed data, while the replay policy is updated to provide the agent with the most useful experiences. The conducted experiments on various continuous control tasks demonstrate the effectiveness of ERO, empirically showing promise in experience replay learning to improve the performance of off-policy reinforcement learning algorithms.

📄 PDF Abstract BibTeX arXiv:1906.08387

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization

2023-02-21 · Yongsheng Mei, Hanhan Zhou, Tian Lan, Guru Venkataramani 외

Experience replay is crucial for off-policy reinforcement learning (RL) methods. By remembering and reusing the experiences from past different policies, experience replay significantly improves the training efficiency a…

Decision Making Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+2

From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay

2026-05-10 · Yanan Xiao, Yixiang Tang, Zechen Feng, Lu Jiang 외 arxiv

While experience replay is essential for data efficiency in reinforcement learning (RL), standard methods treat the replay buffer as a passive memory system, prioritizing samples based on numerical prediction errors rath…

Reinforcement Learning

Variance Reduction Based Experience Replay for Policy Optimization

2026-02-05 · Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy arxiv

Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all pa…

Reinforcement Learning

Accelerating Distributed Deep Reinforcement Learning by In-Network Experience Sampling

2021-10-26 · Masaki Furukawa, Hiroki Matsutani

A computing cluster that interconnects multiple compute nodes is used to accelerate distributed reinforcement learning based on DQN (Deep Q-Network). In distributed reinforcement learning, Actor nodes acquire experiences…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning

2024-02-14 · Jason Yoo, Yunpeng Liu, Frank Wood, Geoff Pleiss

In online continual learning, a neural network incrementally learns from a non-i.i.d. data stream. Nearly all online continual learning methods employ experience replay to simultaneously prevent catastrophic forgetting a…

Continual Learning