MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding

2026-02-26 · Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan arxiv

Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0\% accuracy gain upon the base MLLM, and 1.1\% higher accuracy than strongest baseline method.

📄 PDF Abstract BibTeX arXiv:2602.22932

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Answer Generation

PackLab: A Comprehensive Framework for Developing, Training, and Evaluating MLLMs in Robotic Bin Packing

2026-09-20 · Donghao Zhou, Jia-Hui Pan, Fan Zhang, Xingyuan Bu 외 hf

Robotic bin packing requires long-horizon sequential decision-making, as each object placement affects the available space for subsequent packing. Existing methods primarily rely on hand-crafted geometric heuristics that…

Reinforcement Learning

PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling

2024-10-08 · Xudong Xie, Hao Yan, Liang Yin, Yang Liu 외

Multimodal document understanding is a challenging task to process and comprehend large amounts of textual and visual information. Recent advances in Large Language Models (LLMs) have significantly improved the performan…

document understanding Language Modeling Language Modelling Large Language Model+2

MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs

2026-06-30 · Zhongyang Li, Yaqian Li, Faming Fang, Rinyoichi Takezoe 외 arxiv

Multimodal large language models (MLLMs) typically employ resampling-based projectors to transform dense visual features into a compact token sequence for language modeling. Most existing resamplers adopt a single, fixed…

Multimodal Reasoning

LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

2026-08-26 · Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase, Sam Ade Jacobs 외 arxiv

Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-wei…

A Survey on Agentic Multimodal Large Language Models

2025-10-13 · Huanjin Yao, Ruifei Zhang, Jiaxing Huang, Jingyi Zhang 외 arxiv

With the recent emergence of revolutionary autonomous agentic systems, research community is witnessing a significant shift from traditional static, passive, and domain-specific AI agents toward more dynamic, proactive, …