Temporal Perceiving Video-Language Pre-training
Video-Language Pre-training models have recently significantly improved various multi-modal downstream tasks. Previous dominant works mainly adopt contrastive learning to achieve global feature alignment across modalities. However, the local associations between videos and texts are not modeled, restricting the pre-training models' generality, especially for tasks requiring the temporal video boundary for certain query texts. This work introduces a novel text-video localization pre-text task to enable fine-grained temporal and semantic alignment such that the trained model can accurately perceive temporal boundaries in videos given the text description. Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features. To produce temporal boundaries, frame features in several videos are manually merged into a long video sequence that interacts with a text sequence. With the localization task, our method connects the fine-grained frame representations with the word representations and implicitly distinguishes representations of different instances in the single modality. Notably, comprehensive experimental results show that our method significantly improves the state-of-the-art performance on various benchmarks, covering text-to-video retrieval, video question answering, video captioning, temporal action localization and temporal moment retrieval. The code will be released soon.
Code (0)
등록된 구현이 없습니다.
Tasks
Action LocalizationContrastive LearningMoment RetrievalQuestion AnsweringRetrievalTemporal Action LocalizationText to Video RetrievalVideo CaptioningVideo Question AnsweringVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes
Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal information density. This paper present \textbf{Quicksviewer}, an LMM with n…
MMEVideo MMEVideo UnderstandingGrounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-…
Dense Video CaptioningSentenceTemporal Sentence GroundingVideo Captioning+1A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localization
Current researches on Deepfake forensics often treat detection as a classification task or temporal forgery localization problem, which are usually restrictive, time-consuming, and challenging to scale for large datasets…
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Video understanding relies on perceiving the global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence). To learn these interactions, we apply a mask-then-predict…
Action ClassificationAction RecognitionContrastive LearningVideo UnderstandingVideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
The core challenge in video understanding lies in perceiving dynamic content changes over time. However, multimodal large language models struggle with temporal-sensitive video tasks, which requires generating timestamps…
Instruction FollowingVideo Understanding