Frames2LoRA: Parametric Video Internalization for Vision-Language Models

2026-06-03 · Manan Suri, Sarvesh Baskar, Dinesh Manocha arxiv

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

📄 PDF Abstract BibTeX arXiv:2606.04351

Code (0)

등록된 구현이 없습니다.

Tasks

Video Question Answering Video Summarization

Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization

2026-08-20 · Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou arxiv

Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: convert…

Question Answering

PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization

2026-06-11 · Hao Jiang, Xin Li, Annan Wang, Zhi Yang 외 arxiv

Generative listwise ranking with Large Multimodal Models (LMMs) aims to capture global list context in a single forward pass, but its effectiveness degrades in long-context multimodal scenarios. We identify a recurring f…

Prompt Engineering

Semi-Parametric Video-Grounded Text Generation

2023-01-27 · Sungdong Kim, Jin-Hwa Kim, Jiyoung Lee, Minjoon Seo

Efficient video-language modeling should consider the computational cost because of a large, sometimes intractable, number of video frames. Parametric approaches such as the attention mechanism may not be ideal since its…

Language Modeling Language Modelling Text Generation Video Question Answering+1

Understanding LoRA as Knowledge Memory: An Empirical Analysis

2026-03-01 · Seungju Back, Dongwoo Lee, Naun Kang, Taehee Lee 외 arxiv

Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generat…

Learning through Internalization

2026-06-18 · Nikolaos Tsilivis, Nirmit Joshi, Marko Medvedev, Julia Kempe 외 arxiv

We study internalization processes, by which neural-network-based systems absorb an explicit computational procedure into their own weights, and how they facilitate learning. We investigate how transformers internalize t…