StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs
Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. Prior work shows that shortening the context can sharpen current-scene perception at the expense of long-range recall. To reconcile these abilities, we introduce StreamTTT, which writes long-range history into online-updated fast weights outside the attention context. This leaves a short sliding key-value cache dedicated to recent evidence, mitigating attention dilution. We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus. On OVO-Bench, StreamTTT-4B outperforms SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing. It also remains competitive with the larger SimpleStream-8B on the Real-Time Visual Understanding (RTVU) subset of StreamingBench. Our code will be released.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
Despite significant advances in LLM-driven GUI agents, the field remains constrained by the challenge of reconciling high-fidelity realism with verifiable evaluation accuracy. To address this, we introduce NaturalGAIA, a…
FMRT: Learning Accurate Feature Matching with Reconciliatory Transformer
Local Feature Matching, an essential component of several computer vision tasks (e.g., structure from motion and visual localization), has been effectively settled by Transformer-based methods. However, these methods onl…
Homography EstimationPose EstimationVisual LocalizationModel selection in reconciling hierarchical time series
Model selection has been proven an effective strategy for improving accuracy in time series forecasting applications. However, when dealing with hierarchical time series, apart from selecting the most appropriate forecas…
modelModel SelectionTime SeriesTime Series Analysis+1MTA: Multimodal Task Alignment for BEV Perception and Captioning
Bird's eye view (BEV)-based 3D perception plays a crucial role in autonomous driving applications. The rise of large language models has spurred interest in BEV-based captioning to understand object behavior in the surro…
Autonomous DrivingReconciling Multiple Genes Trees via Segmental Duplications and Losses
Reconciling gene trees with a species tree is a fundamental problem to understand the evolution of gene families. Many existing approaches reconcile each gene tree independently. However, it is well-known that the evolut…