Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations
Probabilistic embeddings have proven useful for capturing polysemous word meanings, as well as ambiguity in image matching. In this paper, we study the advantages of probabilistic embeddings in a cross-modal setting (i.e., text and images), and propose a simple approach that replaces the standard vector point embeddings in extant image-text matching models with probabilistic distributions that are parametrically learned. Our guiding hypothesis is that the uncertainty encoded in the probabilistic embeddings captures the cross-modal ambiguity in the input instances, and that it is through capturing this uncertainty that the probabilistic models can perform better at downstream tasks, such as image-to-text or text-to-image retrieval. Through extensive experiments on standard and new benchmarks, we show a consistent advantage for probabilistic representations in cross-modal retrieval, and validate the ability of our embeddings to capture uncertainty.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Modal RetrievalImage RetrievalImage-text matchingImage to textRetrievalText MatchingSimilar Papers 제목 키워드 기반
Probabilistic Multimodal Representation Learning
Learning multimodal representations is a requirement for many tasks such as image--caption retrieval. Previous work on this problem has only focused on finding good vector representations without any explicit measure of …
Representation LearningRetrievalProbabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
Vision-Language Models (VLMs) learn joint representations by mapping images and text into a shared latent space. However, recent research highlights that deterministic embeddings from standard VLMs often struggle to capt…
Active Learningcross-modal alignmentCross-Modal RetrievalEmbeddings Evaluation+3Probabilistic Embeddings for Cross-Modal Retrieval
Cross-modal retrieval methods build a common representation space for samples from multiple modalities, typically from the vision and the language domains. For images and their captions, the multiplicity of the correspon…
Cross-Modal RetrievalRetrievalEnhancing Partially Relevant Video Retrieval with Robust Alignment Learning
Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos partially relevant to a given query. The core challenge lies in learning robust query-video alignment against spurious semantic correlations ari…
Partially Relevant Video RetrievalVideo AlignmentProbabilistic framework for solving Visual Dialog
In this paper, we propose a probabilistic framework for solving the task of `Visual Dialog'. Solving this task requires reasoning and understanding of visual modality, language modality, and common sense knowledge to ans…
Common Sense ReasoningVisual Dialog