Papers Zero-Shot Video Retrieval
“Zero-Shot Video Retrieval” 태그가 달린 논문 41편 · 필터 해제
VeRVE: Versatile Retrieval for Videos via Unified Embeddings
Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval, fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieva…
Zero-shot Moment RetrievalZero-Shot Video RetrievalMake Your Training Flexible: Towards Deployment-Efficient Video Models
Popular video training methods mainly operate on a fixed number of tokens sampled from a predetermined spatiotemporal grid, resulting in sub-optimal accuracy-computation trade-offs due to inherent video redundancy. They …
Action ClassificationZero-Shot Video RetrievalGramian Multimodal Representation Learning and Alignment
Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligni…
Contrastive LearningRepresentation LearningText RetrievalVideo Classification+3InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive tra…
Action ClassificationAction RecognitionAudio ClassificationContrastive Learning+13vid-TLDR: Training Free Token merging for Light-weight Video Transformer
Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by …
Action RecognitionComputational EfficiencyText RetrievalVideo Question Answering+5OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning
We present a novel multimodal multitask network and associated training algorithm. The method is capable of ingesting data from approximately 12 different modalities namely image video audio text depth point cloud ti…
3D Point Cloud ClassificationAction ClassificationAction RecognitionAudio Classification+6InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical eleme…
Image RetrievalImage-to-Text RetrievalLanguage ModellingLarge Language Model+11HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Instructional videos are a common source for learning text-video or even multimodal representations by leveraging subtitles extracted with automatic speech recognition systems (ASR) from the audio signal in the videos. H…
Automatic Speech RecognitionVideo CaptioningVideo RetrievalZero-Shot Video-Audio Retrieval+1LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vis…
Audio ClassificationContrastive LearningMultimodal Deep LearningScene Classification (unified classes)+11BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning
The recent progress in Large Language Models (LLM) has spurred various advancements in image-language conversation agents, while how to build a proficient video-based dialogue system is still under exploration. Consideri…
GPUVideo-based Generative Performance BenchmarkingVideo-based Generative Performance Benchmarking (Consistency)Video-based Generative Performance Benchmarking (Contextual Understanding)+7VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to es…
Audio captioningAudio-Visual CaptioningAudio-visual Question AnsweringCross-Modal Retrieval+12ImageBind: One Embedding Space To Bind Them All
We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train su…
AllCross-Modal RetrievalMultimodal Deep LearningRetrieval+10Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video…
Action ClassificationAction Recognitioncross-modal alignmentSpatio-Temporal Action Localization+4mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can bene…
Action ClassificationImage ClassificationImage RetrievalTGIF-Frame+7HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-languag…
cross-modal alignmentTGIF-ActionTGIF-FrameTGIF-Transition+6VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners
We explore an efficient approach to establish a foundational video-text model. We present VideoCoCa that maximally reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with m…
Question AnsweringRetrievalText to Video RetrievalVideo Captioning+7InternVideo: General Video Foundation Models via Generative and Discriminative Learning
The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, w…
Action ClassificationAction RecognitionContrastive LearningOpen Set Action Recognition+8Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning
Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language mo…
cross-modal alignmentImage-text RetrievalLanguage ModelingLanguage Modelling+8OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video i…
Action ClassificationAction Recognitioncross-modal alignmentCross-Modal Retrieval+15Clover: Towards A Unified Video-Language Alignment and Fusion Model
Building a universal Video-Language model for solving various video understanding tasks (\emph{e.g.}, text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal,…
Language ModelingLanguage ModellingQuestion AnsweringRetrieval+9