Introspective Experience Replay: Look Back When Surprised

2022-06-07 · Ramnath Kumar, Dheeraj Nagaraj

In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience replay (UER) and prioritized experience replay (PER) have been shown to have sub-optimal convergence and high seed sensitivity respectively. To address these issues, we propose a novel approach called IntrospectiveExperience Replay (IER) that selectively samples batches of data points prior to surprising events. Our method builds upon the theoretically sound reverse experience replay (RER) technique, which has been shown to reduce bias in the output of Q-learning-type algorithms with linear function approximation. However, this approach is not always practical or reliable when using neural function approximation. Through empirical evaluations, we demonstrate that IER with neural function approximation yields reliable and superior performance compared toUER, PER, and hindsight experience replay (HER) across most tasks.

📄 PDF Abstract BibTeX arXiv:2206.03171

Code (1)

google-research/look-back-when-surprised 공식 구현 pytorch

Tasks

Q-Learningreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Prioritized Experience Replay Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

A Deeper Look at Experience Replay

2017-12-04 · Shangtong Zhang, Richard S. Sutton

Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay. It introduces a new hyper-parameter, the memory buffer size, wh…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Saliency Guided Experience Packing for Replay in Continual Learning

2021-09-10 · Gobinda Saha, Kaushik Roy

Artificial learning systems aspire to mimic human intelligence by continually learning from a stream of tasks without forgetting past knowledge. One way to enable such learning is to store past experiences in the form of…

Continual Learning Image Classification

Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains

2018-06-12 · Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson 외

Model-based strategies for control are critical to obtain sample efficient learning. Dyna is a planning paradigm that naturally interleaves learning and planning, by simulating one-step experience to update the action-va…

Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2

2022-08-09 · Zachary Daniels, Aswin Raghavan, Jesse Hostetler, Abrar Rahman 외

One approach to meet the challenges of deep lifelong reinforcement learning (LRL) is careful management of the agent's learning experiences, to learn (without forgetting) and build internal meta-models (of the tasks, env…

Lifelong learning Management reinforcement-learning Reinforcement Learning (RL)+2

Experience Replay Addresses Loss of Plasticity in Continual Learning

2025-03-25 · Jiuqi Wang, Rohan Chandra, Shangtong Zhang

Loss of plasticity is one of the main challenges in continual learning with deep neural networks, where neural networks trained via backpropagation gradually lose their ability to adapt to new tasks and perform significa…

Continual Learning In-Context Learning