Self-ReS: Self-Reflection in Large Vision-Language Models for Long Video Understanding

2025-03-26 · Joao Pereira, Vasco Lopes, David Semedo, Joao Neves

Large Vision-Language Models (LVLMs) demonstrate remarkable performance in short-video tasks such as video question answering, but struggle in long-video understanding. The linear frame sampling strategy, conventionally used by LVLMs, fails to account for the non-linear distribution of key events in video data, often introducing redundant or irrelevant information in longer contexts while risking the omission of critical events in shorter ones. To address this, we propose SelfReS, a non-linear spatiotemporal self-reflective sampling method that dynamically selects key video fragments based on user prompts. Unlike prior approaches, SelfReS leverages the inherently sparse attention maps of LVLMs to define reflection tokens, enabling relevance-aware token selection without requiring additional training or external modules. Experiments demonstrate that SelfReS can be seamlessly integrated into strong base LVLMs, improving long-video task accuracy and achieving up to 46% faster inference speed within the same GPU memory budget.

📄 PDF Abstract BibTeX arXiv:2503.20362

Code (0)

등록된 구현이 없습니다.

Tasks

GPUQuestion Answering Video Question AnsweringVideo Understanding

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

BASE 설명 없음

Vision-Language Models Can Self-Improve Reasoning via Reflection

2024-10-30 · Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang 외

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT re…

Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

2026-07-02 · Liyan Tang, Fangcong Yin, Greg Durrett arxiv

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting pr…

Reinforcement Learning Visual Grounding

SRVAU-R1: Enhancing Video Anomaly Understanding via Reflection-Aware Learning

2026-02-01 · Zihao Zhao, Shengting Cao, Muchao Ye arxiv

Multi-modal large language models (MLLMs) have demonstrated significant progress in reasoning capabilities and shown promising effectiveness in video anomaly understanding (VAU) tasks. However, existing MLLM-based approa…

SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning

2025-06-02 · Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang 외

Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle with complex problems requiring explicit self-reflection and self-correction, especially compared to their…

Multimodal Reasoning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?

2026-05-09 · Filippo Ziliotto, Ciro Beneduce, Bruno Lepri, Luciano Serafini 외 arxiv

In the animal kingdom, mirror self-recognition is a canonical probe of higher-order cognition, emerging only in some species. We ask whether an analogous functional capability emerges in embodied vision-language model (V…