Contrastive Language Video Time Pre-training
We introduce LAVITI, a novel approach to learning language, video, and temporal representations in long-form videos via contrastive learning. Different from pre-training on video-text pairs like EgoVLP, LAVITI aims to align language, video, and temporal features by extracting meaningful moments in untrimmed videos. Our model employs a set of learnable moment queries to decode clip-level visual, language, and temporal features. In addition to vision and language alignment, we introduce relative temporal embeddings (TE) to represent timestamps in videos, which enables contrastive learning of time. Significantly different from traditional approaches, the prediction of a particular timestamp is transformed by computing the similarity score between the predicted TE and all TEs. Furthermore, existing approaches for video understanding are mainly designed for short videos due to high computational complexity and memory footprint. Our method can be trained on the Ego4D dataset with only 8 NVIDIA RTX-3090 GPUs in a day. We validated our method on CharadesEgo action recognition, achieving state-of-the-art results.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionContrastive LearningMoment QueriesVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
BERT-type structure has led to the revolution of vision-language pre-training and the achievement of state-of-the-art results on numerous vision-language downstream tasks. Existing solutions dominantly capitalize on the …
Cross-Modal RetrievalDecoderDenoisingLanguage Modeling+6Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
Multi-channel video-language retrieval require models to understand information from different channels (e.g. video$+$question, video$+$speech) to correctly link a video with a textual response or query. Fortunately, con…
RetrievalSentenceSentence EmbeddingsVideo Question AnsweringUnderstanding Chinese Video and Language via Contrastive Multimodal Pre-Training
The pre-trained neural models have recently achieved impressive performances in understanding multimodal content. However, it is still very challenging to pre-train neural models for video and language understanding, esp…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+1DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generation
Flow-matching video generators produce temporally coherent, high-fidelity outputs yet routinely violate elementary physics because their reconstruction objectives penalize per-frame deviations without distinguishing phys…
Contrastive LearningVideo GenerationExpectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
Most video-and-language representation learning approaches employ contrastive learning, e.g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video p…
Contrastive LearningRepresentation LearningRetrievalVideo Captioning+3