VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model. We present VideoCoCa that maximally reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules, we find that the generative attentional pooling and contrastive attentional pooling layers in CoCa are instantly adaptable to flattened frame embeddings, yielding state-of-the-art results on zero-shot video classification and zero-shot text-to-video retrieval. Furthermore, we explore lightweight finetuning on top of VideoCoCa, and achieve strong results on video question-answering and video captioning.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringRetrievalText to Video RetrievalVideo CaptioningVideo ClassificationVideo Question AnsweringVideo RetrievalVideo to Text RetrievalVisual Question Answering (VQA)Zero-Shot Action RecognitionZero-Shot Video RetrievalSimilar Papers 제목 키워드 기반
Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
Large-scale text-to-image diffusion models achieve unprecedented success in image generation and editing. However, how to extend such success to video editing is unclear. Recent initial attempts at video editing require …
Video AlignmentVideo EditingMILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval
Dominant pre-training work for video-text retrieval mainly adopt the "dual-encoder" architectures to enable efficient retrieval, where two separate encoders are used to contrast global video and text representations, but…
Action RecognitionRetrievalText RetrievalText to Video Retrieval+5Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle thi…
Fill MaskLanguage ModelingLanguage ModellingMasked Language Modeling+8Telling Stories for Common Sense Zero-Shot Action Recognition
Video understanding has long suffered from reliance on large labeled datasets, motivating research into zero-shot learning. Recent progress in language modeling presents opportunities to advance zero-shot video analysis,…
Action RecognitionArticlesCommon Sense ReasoningLanguage Modeling+6SGCap: Decoding Semantic Group for Zero-shot Video Captioning
Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs, which remains underexplored. Existing zero-shot image captioning methods typically adopt a text-…
Video CaptioningImage Captioning