Zero-Shot Video Captioning with Evolving Pseudo-Tokens
We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model. The matching score is used to steer the language model toward generating a sentence that has a high average matching score to a subset of the video frames. Unlike zero-shot image captioning methods, our work considers the entire sentence at once. This is achieved by optimizing, during the generation process, part of the prompt from scratch, by modifying the representation of all other tokens in the prompt, and by repeating the process iteratively, gradually improving the specificity and comprehensiveness of the generated sentence. Our experiments show that the generated captions are coherent and display a broad range of real-world knowledge. Our code is available at: https://github.com/YoadTew/zero-shot-video-to-text
Code (1)
Tasks
Image CaptioningImage-text matchingLanguage ModelingLanguage ModellingSentenceSpecificityText MatchingVideo CaptioningWorld KnowledgeMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zero-shot Natural Language Video Localization
Understanding videos to localize moments with natural language often requires large expensive annotated video regions paired with language queries. To eliminate the annotation costs, we make a first attempt to train a na…
Image CaptioningSGCap: Decoding Semantic Group for Zero-shot Video Captioning
Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs, which remains underexplored. Existing zero-shot image captioning methods typically adopt a text-…
Video CaptioningImage CaptioningLearning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Although promising results have been achieved in video captioning, existing models are limited to the fixed inventory of activities in the training corpus, and do not generalize to open vocabulary scenarios. Here we intr…
Mixture-of-ExpertsVideo CaptioningFine-Grained Video Captioning through Scene Graph Consolidation
Recent advances in visual language models (VLMs) have significantly improved image captioning, but extending these gains to video understanding remains challenging due to the scarcity of fine-grained video captioning dat…
Caption GenerationImage CaptioningVideo CaptioningVideo UnderstandingVid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with spec…
Dense Video CaptioningLanguage ModelingLanguage ModellingSentence+1