Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval
In a retrieval system, simultaneously achieving search accuracy and efficiency is inherently challenging. This challenge is particularly pronounced in partially relevant video retrieval (PRVR), where incorporating more diverse context representations at varying temporal scales for each video enhances accuracy but increases computational and memory costs. To address this dichotomy, we propose a prototypical PRVR framework that encodes diverse contexts within a video into a fixed number of prototypes. We then introduce several strategies to enhance text association and video understanding within the prototypes, along with an orthogonal objective to ensure that the prototypes capture a diverse range of content. To keep the prototypes searchable via text queries while accurately encoding video contexts, we implement cross- and uni-modal reconstruction tasks. The cross-modal reconstruction task aligns the prototypes with textual features within a shared space, while the uni-modal reconstruction task preserves all video contexts during encoding. Additionally, we employ a video mixing technique to provide weak guidance to further align prototypes and associated textual representations. Extensive evaluations on TVR, ActivityNet-Captions, and QVHighlights validate the effectiveness of our approach without sacrificing efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Partially Relevant Video RetrievalRetrievalVideo RetrievalVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalized Category Discovery in Event-Centric Contexts: Latent Pattern Mining with LLMs
Generalized Category Discovery (GCD) aims to classify both known and novel categories using partially labeled data that contains only known classes. Despite achieving strong performance on existing benchmarks, current te…
Equilibrium contrastive learning for imbalanced image classification
Contrastive learning (CL) is a predominant technique in image classification, but they showed limited performance with an imbalanced dataset. Recently, several supervised CL methods have been proposed to promote an ideal…
Contrastive LearningImage ClassificationToward Real-Time Image Annotation Using Marginalized Coupled Dictionary Learning
In most image retrieval systems, images include various high-level semantics, called tags or annotations. Virtually all the state-of-the-art image annotation methods that handle imbalanced labeling are search-based techn…
Dictionary LearningImage RetrievalRetrievalGeneralized Unbiased Scene Graph Generation
Existing Unbiased Scene Graph Generation (USGG) methods only focus on addressing the predicate-level imbalance that high-frequency classes dominate predictions of rare ones, while overlooking the concept-level imbalance.…
Graph GenerationScene Graph GenerationUnbiased Scene Graph GenerationGenerating Real-Time Strategy Game Units Using Search-Based Procedural Content Generation and Monte Carlo Tree Search
Real-Time Strategy (RTS) game unit generation is an unexplored area of Procedural Content Generation (PCG) research, which leaves the question of how to automatically generate interesting and balanced units unanswered. C…