Memory Consolidation Enables Long-Context Video Understanding

2024-02-08 · Ivana Balažević, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, Olivier J. Hénaff

Most transformer-based video encoders are limited to short temporal contexts due to their quadratic complexity. While various attempts have been made to extend this context, this has often come at the cost of both conceptual and computational complexity. We propose to instead re-purpose existing pre-trained video transformers by simply fine-tuning them to attend to memories derived non-parametrically from past activations. By leveraging redundancy reduction, our memory-consolidated vision transformer (MC-ViT) effortlessly extends its context far into the past and exhibits excellent scaling behavior when learning from longer videos. In doing so, MC-ViT sets a new state-of-the-art in long-context video understanding on EgoSchema, Perception Test, and Diving48, outperforming methods that benefit from orders of magnitude more parameters.

📄 PDF Abstract BibTeX arXiv:2402.05861

Code (0)

등록된 구현이 없습니다.

Tasks

EgoSchemaVideo Understanding

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

$\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation

2025-01-31 · Saul Santos, António Farinhas, Daniel C. McNamee, André F. T. Martins

Current video-language models struggle with long-video understanding due to limited context lengths and reliance on sparse frame subsampling, often leading to information loss. This paper introduces $\infty$-Video, which…

Question Answering Video Question Answering Video Understanding

TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents

2026-01-06 · Kai Li, Xuanqing Yu, Ziyi Ni, Yi Zeng 외 arxiv

Long-horizon conversational agents have to manage ever-growing interaction histories that quickly exceed the finite context windows of large language models (LLMs). Existing memory frameworks provide limited support for …

LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation

2026-08-13 · Dongfang Li, Zixuan Liu, Junmai Wang, Jiahe Huang 외 arxiv

Long-horizon LLM agents must preserve information from past interactions to support future tasks. Existing memory systems typically rely on eager consolidation, invoking LLMs after each interaction to extract, summarize,…

Boundary Detection

FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion

2026-06-09 · Yu Lu, Junjie Yang, Piotr Koniusz, YuXin Song 외 arxiv

Autoregressive video generators synthesize long videos by generating successive temporal segments, but their historical KV cache grows with video length. Existing bounded-cache methods reduce this cost with local windows…

Episodic-Semantic Memory Architecture for Long-Horizon Scientific Agents

2026-05-17 · Nikola Milosevic arxiv

As Large Language Models (LLMs) evolve into persistent scientific collaborators, context window saturation has emerged as a critical bottleneck. Scientific workflows involving iterative data analysis and hypothesis refin…