VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.
Code (2)
Tasks
Action LocalizationAction SegmentationLong Video Retrieval (Background Removed)RetrievalTemporal Action LocalizationTemporal Relation ExtractionVideo RetrievalZero-Shot Video RetrievalSimilar Papers 제목 키워드 기반
VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models
Contrastive Language-Image Pre-training (CLIP) has been widely studied and applied in numerous applications. However, the emphasis on brief summary texts during pre-training prevents CLIP from understanding long descript…
Hallucinationtext similarityVideo DescriptionVideo RetrievalTest of Time: Instilling Video-Language Models with a Sense of Time
Modelling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language mo…
Video-Text RetrievalVideo UnderstandingVideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model. We present VideoCoCa that maximally reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with m…
Question AnsweringRetrievalText to Video RetrievalVideo Captioning+7Contrastive Video-Language Learning with Fine-grained Frame Sampling
Despite recent progress in video and language representation learning, the weak or sparse correspondence between the two modalities remains a bottleneck in the area. Most video-language models are trained via pair-level …
Question AnsweringRepresentation LearningRetrievalVideo Question Answering+1Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization
Contrastive Language-Image Pretraining (CLIP) has demonstrated impressive zero-shot learning abilities for image understanding, yet limited effort has been made to investigate CLIP for zero-shot video recognition. We int…
Action RecognitionContinual LearningVideo RecognitionZero-Shot Learning