Papers Video-Text Retrieval
“Video-Text Retrieval” 태그가 달린 논문 125편 · 필터 해제
V$^2$Dial: Unification of Video and Visual Dialog via Multimodal Experts
We present V$^2$Dial - a novel expert-based model specifically geared towards simultaneously handling image and video input data for multimodal conversational tasks. Current multimodal models primarily focus on simpler t…
Contrastive LearningText RetrievalVideo-Text RetrievalVisual Dialog+1Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding
Multimodal foundation models (MFMs) have demonstrated significant success in tasks such as visual captioning, question answering, and image-text retrieval. However, these models face inherent limitations due to their fin…
Image CaptioningImage-text RetrievalQuestion AnsweringRetrieval+4Expertized Caption Auto-Enhancement for Video-Text Retrieval
Video-text retrieval has been stuck in the information mismatch caused by personalized and inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders an effective cross-…
Caption GenerationRetrievalText RetrievalVideo-Text RetrievalRethinking Noisy Video-Text Retrieval via Relation-aware Alignment
Video-Text Retrieval (VTR) is a core task in multi-modal understanding, drawing growing attention from both academia and industry in recent years. While numerous VTR methods have achieved success, most of them assume…
RelationRetrievalText RetrievalVideo-Text RetrievalV^2Dial: Unification of Video and Visual Dialog via Multimodal Experts
We present V2Dial - a novel expert-based model specifically geared towards simultaneously handling image and video input data for multimodal conversational tasks. Current multimodal models primarily focus on simpler …
Contrastive LearningText RetrievalVideo-Text RetrievalVisual Dialog+1CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval
Video understanding, including video captioning and retrieval, is still a great challenge for video-language models (VLMs). The existing video retrieval and caption benchmarks only include short descriptions, limits thei…
RetrievalText RetrievalText to Video RetrievalVideo Captioning+3Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
Cross-modal (e.g. image-text, video-text) retrieval is an important task in information retrieval and multimodal vision-language understanding field. Temporal understanding makes video-text retrieval more challenging tha…
Image-text RetrievalInformation RetrievalRetrievalText Retrieval+1CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectives
Grounding the instruction in the environment is a key step in solving language-guided goal-reaching reinforcement learning problems. In automated reinforcement learning, a key concern is to enhance the model's ability to…
reinforcement-learningReinforcement LearningRetrievalSystematic Generalization+2Beyond Coarse-Grained Matching in Video-Text Retrieval
Video-text retrieval has seen significant advancements, yet the ability of models to discern subtle differences in captions still requires verification. In this paper, we introduce a new approach for fine-grained evaluat…
RetrievalText RetrievalVideo-Text RetrievalText Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval
Text-video retrieval (TVR) has seen substantial advancements in recent years, fueled by the utilization of pre-trained models and large language models (LLMs). Despite these advancements, achieving accurate matching in T…
RetrievalText RetrievalVideo RetrievalVideo-Text RetrievalNAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality
We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. Composition understanding becomes particularly challenging for video data si…
RetrievalText RetrievalVideo RetrievalVideo-Text RetrievalLearning Video Context as Interleaved Multimodal Sequences
Narrative videos, such as movies, pose significant challenges in video understanding due to their rich contexts (characters, dialogues, storylines) and diverse demands (identify who, relationship, and reason). In this pa…
Language ModelingLanguage ModellingQuestion AnsweringText Retrieval+5Video-Language Alignment via Spatio-Temporal Graph Transformer
Video-language alignment is a crucial multi-modal task that benefits various downstream applications, e.g., video-text retrieval and video question answering. Existing methods either utilize multi-modal information in vi…
Contrastive LearningQuestion AnsweringRetrievalText Retrieval+2EA-VTR: Event-Aware Video-Text Retrieval
Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and th…
Action RecognitionContrastive Learningcross-modal alignmentMoment Retrieval+6Multi-Scale Temporal Difference Transformer for Video-Text Retrieval
Currently, in the field of video-text retrieval, there are many transformer-based methods. Most of them usually stack frame features and regrade frames as tokens, then use transformers for video temporal modeling. Howeve…
RetrievalText RetrievalVideo-Text RetrievalDiving Deep into the Motion Representation of Video-Text Models
Videos are more informative than images because they capture the dynamics of the scene. By representing motion in videos, we can capture dynamic activities. In this work, we introduce GPT-4 generated motion descriptions …
RetrievalText RetrievalVideo-Text RetrievalHENASY: Learning to Assemble Scene-Entities for Egocentric Video-Language Model
Current video-language models (VLMs) rely extensively on instance-level alignment between video and language modalities, which presents two major limitations: (1) visual reasoning disobeys the natural perception that hum…
Action RecognitionActivity RecognitionDecoderLanguage Modeling+6Uncertainty-aware sign language video retrieval with probability distribution modeling
Sign language video retrieval plays a key role in facilitating information access for the deaf community. Despite significant advances in video-text retrieval, the complexity and inherent uncertainty of sign language pre…
RetrievalSign Language RetrievalText RetrievalVideo Retrieval+1An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval
CLIP4Clip model transferred from the CLIP has been the de-factor standard to solve the video clip retrieval task from frame-level input, triggering the surge of CLIP4Clip-based models in the video-text retrieval domain. …
RetrievalText RetrievalVideo-Text RetrievalRETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioning
Despite the significant progress of fully-supervised video captioning, zero-shot methods remain much less explored. In this paper, we propose a novel zero-shot video captioning framework named Retrieval-Enhanced Test-Tim…
Image-text matchingRetrievalTest-time AdaptationText Generation+5