Memory Lens: How Much Memory Does an Agent Use?

2016-11-21 · Christoph Dann, Katja Hofmann, Sebastian Nowozin

We propose a new method to study the internal memory used by reinforcement learning policies. We estimate the amount of relevant past information by estimating mutual information between behavior histories and the current action of an agent. We perform this estimation in the passive setting, that is, we do not intervene but merely observe the natural behavior of the agent. Moreover, we provide a theoretical justification for our approach by showing that it yields an implementation-independent lower bound on the minimal memory capacity of any agent that implement the observed policy. We demonstrate our approach by estimating the use of memory of DQN policies on concatenated Atari frames, demonstrating sharply different use of memory across 49 games. The study of memory as information that flows from the past to the current action opens avenues to understand and improve successful reinforcement learning algorithms.

📄 PDF Abstract BibTeX arXiv:1611.06928

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

MemLens: A Value-Aware Memory Management System with Interactive Analytics for LLM-based Agents

2026-07-28 · Shuyue Wei, Chang Liu, Zimu Zhou, Yongxin Tong 외 arxiv

Recently, memory management has become a key infrastructure for LLM-based agents, as it directly affects long-horizon reasoning, personalized responses, and knowledge reuse. However, existing LLM memory systems typically…

MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

2026-05-14 · Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie 외 arxiv

Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no exis…

Information Extraction Visual Grounding

The Effects of Memory Replay in Reinforcement Learning

2017-10-18 · Ruishan Liu, James Zou

Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despit…

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

2026-08-05 · Yushi Sun, Yanjie Zhang hf

Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an agent must reconcile a confident memory claim with a contrad…

Visual Grounding

Shared Selective Persistent Memory for Agentic LLM Systems

2026-07-10 · Sanjana Pedada, Aditya Dhavala, Neelraj Patil arxiv

Agentic LLM systems that generate code through multi-turn tool use face a fundamental context problem: each session starts from zero, discarding the domain constraints, data schemas, tool configurations, and output prefe…