Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos, identify the most relevant information based on contextual cues from a given question, and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However, for video QA, an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work, we investigate diverse temporal modeling techniques to integrate with MLLMs, aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former, a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVideo Question AnsweringVideo UnderstandingSimilar Papers 제목 키워드 기반
VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding
Recent advancements in language-model-based video understanding have been progressing at a remarkable pace, spurred by the introduction of Large Language Models (LLMs). However, the focus of prior research has been predo…
Language ModelingLanguage ModellingQuestion AnsweringVideo Question Answering+1MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly f…
Multimodal ReasoningOmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visu…
Self-Supervised LearningContrastive LearningVisual ReasoningiPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
Most prior art in visual understanding relies solely on analyzing the "what" (e.g., event recognition) and "where" (e.g., event localization), which in some cases, fails to describe correct contextual relationships betwe…
Common Sense ReasoningDense Video CaptioningMachine TranslationQuestion Answering+2VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models
We propose VideoPerceiver, a novel video multimodal large language model (VMLLM) that enhances fine-grained perception in video understanding, addressing VMLLMs' limited ability to reason about brief actions in short cli…
Reinforcement LearningAction Understanding