LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

2026-08-26 · Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase, Sam Ade Jacobs, Mathis Bode, Mohamed Elhoseiny arxiv

Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.

📄 PDF Abstract BibTeX arXiv:2608.25729

Code (0)

등록된 구현이 없습니다.

LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

2024-10-22 · Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu 외

Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To a…

Token Reduction Video Question Answering Video Understanding Zero-Shot Video Question Answer

LongViTU: Instruction Tuning for Long-Form Video Understanding

2025-01-09 · Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan 외

This paper introduces LongViTU, a large-scale (~121k QA pairs, ~900h videos), automatically generated dataset for long-form video understanding. We propose a systematic approach that organizes videos into a hierarchical …

EgoSchema Form Instruction Following Video Understanding

CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding

2026-01-29 · Jiahao Huo, Yu Huang, Yibo Yan, Ye Pan 외 arxiv

Although Multimodal Large Language Models (MLLMs) have shown remarkable potential in Visual Document Retrieval (VDR) through generating high-quality multi-vector embeddings, the substantial storage overhead caused by rep…

A Causality-aware Infer-diagnose-refine Framework for Test-time Modality Adaptation in VLA Models

2026-07-28 · Haoyu Zhang, Yuwei Wu, Jin Chen, Gao Zhi 외 arxiv

Vision-language-action (VLA) models predict sequential actions to execute tasks specified by language instructions, conditioned on visual observations and proprioceptive states. However, how to fuse modalities in VLA mod…

Robot Manipulation

BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models

2026-01-05 · Sunny Gupta, Shounak Das, Amit Sethi arxiv

Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address …

Zero-shot Generalization Test-time Adaptation