CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval
This paper tackles a recently proposed Video Corpus Moment Retrieval task. This task is essential because advanced video retrieval applications should enable users to retrieve a precise moment from a large video corpus. We propose a novel CONtextual QUery-awarE Ranking~(CONQUER) model for effective moment localization and ranking. CONQUER explores query context for multi-modal fusion and representation learning in two different steps. The first step derives fusion weights for the adaptive combination of multi-modal video content. The second step performs bi-directional attention to tightly couple video and query as a single joint representation for moment localization. As query context is fully engaged in video representation learning, from feature fusion to transformation, the resulting feature is user-centered and has a larger capacity in capturing multi-modal signals specific to query. We conduct studies on two datasets, TVR for closed-world TV episodes and DiDeMo for open-world user-generated videos, to investigate the potential advantages of fusing video and query online as a joint representation for moment retrieval.
Code (1)
Tasks
Corpus Video Moment RetrievalMoment Retrievalorpus Video Moment RetrievalRepresentation LearningRetrievalrpus Video Moment RetrievalVideo Corpus Moment RetrievalVideo RetrievalSimilar Papers 제목 키워드 기반
ConQueR: Contextualized Query Reduction using Search Logs
Query reformulation is a key mechanism to alleviate the linguistic chasm of query in ad-hoc retrieval. Among various solutions, query reduction effectively removes extraneous terms and specifies concise user intent from …
Language ModelingLanguage ModellingRetrievalTerm ExtractionSee, Rank, and Filter: Important Word-Aware Clip Filtering via Scene Understanding for Moment Retrieval and Highlight Detection
Video moment retrieval (MR) and highlight detection (HD) with natural language queries aim to localize relevant moments and key highlights in a video clips. However, existing methods overlook the importance of individual…
Natural Language QueriesHighlight DetectionScene UnderstandingMoment RetrievalCompact Token Representations with Contextual Quantization for Efficient Document Re-ranking
Transformer based re-ranking models can achieve high search relevance through context-aware soft matching of query tokens with document tokens. To alleviate runtime complexity of such inference, previous work has adopted…
QuantizationRe-RankingCompact Token Representations with Contextual Quantization for Efficient Document Re-ranking
Transformer based re-ranking models can achieve high search relevance through context-aware soft matching of query tokens with document tokens. To alleviate runtime complexity of such inference, previous work has adopte…
QuantizationRe-RankingCONQUER: Context-Aware Representation with Query Enhancement for Text-Based Person Search
Text-Based Person Search (TBPS) aims to retrieve pedestrian images from large galleries using natural language descriptions. This task, essential for public safety applications, is hindered by cross-modal discrepancies a…
Person Search