Analysis of Stochastic Processes through Replay Buffers

2022-06-26 · Shirli Di Castro Shashua, Shie Mannor, Dotan Di-Castro

Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.

📄 PDF Abstract BibTeX arXiv:2206.12848

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ARROW: Augmented Replay for RObust World models

2026-03-12 · Abdulaziz Alyahya, Abdallah Al Siyabi, Markus R. Ernst, Luke Yang 외 arxiv

Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones with the goal of improving performance in both past and future tasks. Most existing approaches rely on mode…

Reinforcement Learning

B2RL: An open-source Dataset for Building Batch Reinforcement Learning

2022-09-30 · Hsin-Yu Liu, Xiaohan Fu, Bharathan Balaji, Rajesh Gupta 외

Batch reinforcement learning (BRL) is an emerging research area in the RL community. It learns exclusively from static datasets (i.e. replay buffers) without interaction with the environment. In the offline settings, exi…

Management reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning

2023-12-05 · Pankayaraj Pathmanathan, Natalia Díaz-Rodríguez, Javier Del Ser

In this work, we investigate the means of using curiosity on replay buffers to improve offline multi-task continual reinforcement learning when tasks, which are defined by the non-stationarity in the environment, are non…

Boundary Detection reinforcement-learning Reinforcement Learning

Heads collapse, features stay: Why Replay needs big buffers

2025-12-08 · Giulia Lanzillotta, Damiano Meier, Thomas Hofmann arxiv

A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We formalize this distinction as the gap betw…

Out-of-Distribution Detection Continual Learning

Streaming Linear System Identification with Reverse Experience Replay

2021-03-10 · NeurIPS 2021 12 · Prateek Jain, Suhas S Kowshik, Dheeraj Nagaraj, Praneeth Netrapalli

We consider the problem of estimating a linear time-invariant (LTI) dynamical system from a single trajectory via streaming algorithms, which is encountered in several applications including reinforcement learning (RL) a…

Reinforcement Learning (RL)Time Series Analysis