HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding
Despite advancements in multimodal large language models (MLLMs), current approaches struggle in medium-to-long video understanding due to frame and context length limitations. As a result, these models often depend on frame sampling, which risks missing key information over time and lacks task-specific relevance. To address these challenges, we introduce HierarQ, a task-aware hierarchical Q-Former based framework that sequentially processes frames to bypass the need for frame sampling, while avoiding LLM's context length limitations. We introduce a lightweight two-stream language-guided feature modulator to incorporate task awareness in video understanding, with the entity stream capturing frame-level object information within a short context and the scene stream identifying their broader interactions over longer period of time. Each stream is supported by dedicated memory banks which enables our proposed Hierarchical Querying transformer (HierarQ) to effectively capture short and long-term context. Extensive evaluations on 10 video benchmarks across video understanding, question answering, and captioning tasks demonstrate HierarQ's state-of-the-art performance across most datasets, proving its robustness and efficiency for comprehensive video analysis. All code will be made available upon acceptance.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVideo UnderstandingSimilar Papers 제목 키워드 기반
EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification
Hybrid vision architectures combining Transformers and CNNs have significantly advanced image classification, but they usually do so at significant computational cost. We introduce EVCC (Enhanced Vision Transformer-ConvN…
Image ClassificationMulti-Task LearningMISSFormer: An Effective Medical Image Segmentation Transformer
The CNN-based methods have achieved impressive results in medical image segmentation, but it failed to capture the long-range dependencies due to the inherent locality of convolution operation. Transformer-based methods …
Cardiac SegmentationImage SegmentationMedical Image SegmentationSegmentation+1RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation
Endowing chatbots with a consistent persona is essential to an engaging conversation, yet it remains an unresolved challenge. In this work, we propose a new retrieval-enhanced approach for personalized response generatio…
DecoderResponse GenerationRetrievalStructure-Enhanced Pop Music Generation via Harmony-Aware Learning
Pop music generation has always been an attractive topic for both musicians and scientists for a long time. However, automatically composing pop music with a satisfactory structure is still a challenging issue. In this p…
FormMusic GenerationDomain-Agnostic Causal-Aware Audio Transformer for Infant Cry Classification
Accurate and interpretable classification of infant cry paralinguistics is essential for early detection of neonatal distress and clinical decision support. However, many existing deep learning methods rely on correlatio…
Representation LearningDomain GeneralizationMulti-Task Learning