FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

2026-07-28 · Ghazal Kaviani, Ghassan AlRegib arxiv

Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.

📄 PDF Abstract BibTeX arXiv:2607.25266

Code (0)

등록된 구현이 없습니다.

Tasks

Question Answering

Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models

2025-09-07 · Kefan Cao, Shuaicheng Wu arxiv

Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonali…

Multi-Task Learning Continual Learning

Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment

2026-04-13 · Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour arxiv

Robots must verbalize their past experiences when users ask "Where did you put my keys?" or "Why did the task fail?" Yet maintaining life-long episodic memory (EM) from continuous multimodal perception quickly exceeds st…

$\textbf{AGT$^{AO}$}$: Robust and Stabilized LLM Unlearning via Adversarial Gating Training with Adaptive Orthogonality

2026-02-02 · Pengyu Li, Lingling Zhang, Zhitao Gao, Yanrui Wu 외 arxiv

While Large Language Models (LLMs) have achieved remarkable capabilities, they unintentionally memorize sensitive data, posing critical privacy and security risks. Machine unlearning is pivotal for mitigating these risks…

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation

2026-02-10 · Brady Steele arxiv

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory character…

parameter-efficient fine-tuning Continual Learning

Novel Memory Forgetting Techniques for Autonomous AI Agents: Balancing Relevance and Efficiency

2026-04-02 · Payal Fofadiya, Sunil Tiwari arxiv

Long-horizon conversational agents require persistent memory for coherent reasoning, yet uncontrolled accumulation causes temporal decay and false memory propagation. Benchmarks such as LOCOMO and LOCCO report performanc…