Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

2023-05-03 · Muhammad Burhan Hafez, Tilman Immisch, Tom Weber, Stefan Wermter

Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. Replay Memories are a common solution to the problem, decorrelating and shuffling old and new training samples. They naively store state transitions as they come in, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our paper shows that map-based experience replay allows for significant memory reduction with only small performance decreases.

📄 PDF Abstract BibTeX arXiv:2305.02054

Code (1)

tilmanimmisch/gwrr 공식 구현 pytorch

Tasks

Deep Reinforcement Learning

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning

2024-02-14 · Jason Yoo, Yunpeng Liu, Frank Wood, Geoff Pleiss

In online continual learning, a neural network incrementally learns from a non-i.i.d. data stream. Nearly all online continual learning methods employ experience replay to simultaneously prevent catastrophic forgetting a…

Continual Learning

Selective Experience Replay for Lifelong Learning

2018-02-28 · David Isele, Akansel Cosgun

Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an e…

Deep Reinforcement Learning Lifelong learning Reinforcement Learning

Federated Learning Under Temporal Drift -- Mitigating Catastrophic Forgetting via Experience Replay

2026-01-19 · Sahasra Kokkula, Daniel David, Aaditya Baruah arxiv

Federated Learning struggles under temporal concept drift where client data distributions shift over time. We demonstrate that standard FedAvg suffers catastrophic forgetting under seasonal drift on Fashion-MNIST, with a…

Federated Learning

AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

2023-08-07 · Xingyu Li, Bo Tang, Haifeng Li

Continual lifelong learning is an machine learning framework inspired by human learning, where learners are trained to continuously acquire new knowledge in a sequential manner. However, the non-stationary nature of stre…

class-incremental learning Class Incremental Learning Incremental Learning Lifelong learning

Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation

2022-05-22 · Qingfeng Lan, Yangchen Pan, Jun Luo, A. Rupam Mahmood

Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)