AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

2026-03-30 · Haozhe Qi, Kevin Qu, Mahdi Rad, Rui Wang, Alexander Mathis, Marc Pollefeys arxiv

Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token

📄 PDF Abstract BibTeX arXiv:2603.28696

Code (0)

등록된 구현이 없습니다.

Adaptive Keyframe Sampling for Long Video Understanding

2025-01-01 · CVPR 2025 1 · Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian 외

Multimodal large language models (MLLMs) have enabled open-world visual understanding by injecting visual input as extra tokens into large language models (LLMs) as contexts. However, when the visual input changes fr…

Video Understanding

Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models

2026-09-15 · Zhenbin Wang, Lei Zhang, Lituan Wang, Wei Huang 외 arxiv

Multimodal large language models (MLLMs) can answer knowledge-intensive visual questions by combining visual evidence from images with facts retrieved from external sources. However, MLLMs may overlook relevant evidence …

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

2026-06-30 · Yuhao Wang, Mu Qiao, Haiwen Diao, Yunzhi Zhuge 외 arxiv

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attent…

Entropy-UID: A Method for Optimizing Information Density

2025-02-20 · Xinpeng Shou

Balanced and efficient information flow is essential for optimizing language generation models. In this work, we propose Entropy-UID, a new token selection method that balances entropy and Uniform Information Density (UI…

Text Generation

Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

2026-06-30 · Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang 외 arxiv

Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable…

Reinforcement Learning