Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings; at best, it would find an average representation of different meanings. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. Most existing work on cross-modal retrieval focuses on image-text data. Here, we also tackle a more challenging case of video-text retrieval. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset.
Code (1)
Tasks
Cross-Modal RetrievalMultiple Instance LearningRetrievalSentenceText RetrievalVideo-Text RetrievalSimilar Papers 제목 키워드 기반
How does BERT capture semantics? A closer look at polysemous words
The recent paradigm shift to contextual word embeddings has seen tremendous success across a wide range of down-stream tasks. However, little is known on how the emergent relation of context and semantics manifests geome…
Semanticity predictionSemantic SimilarityWord EmbeddingsWord Sense Disambiguation+2Multi Sense Embeddings from Topic Models
Distributed word embeddings have yielded state-of-the-art performance in many NLP tasks, mainly due to their success in capturing useful semantic information. These representations assign only a single vector to each wor…
Topic ModelsWord EmbeddingsWord SimilarityUncertainty-based Cross-Modal Retrieval with Probabilistic Representations
Probabilistic embeddings have proven useful for capturing polysemous word meanings, as well as ambiguity in image matching. In this paper, we study the advantages of probabilistic embeddings in a cross-modal setting (i.e…
Cross-Modal RetrievalImage RetrievalImage-text matchingImage to text+2Chinese Word Sense Embedding with SememeWSD and Synonym Set
Word embedding is a fundamental natural language processing task which can learn feature of words. However, most word embedding methods assign only one vector to a word, even if polysemous words have multi-senses. To add…
Semantic SimilaritySemantic Textual SimilarityWord Sense DisambiguationTowards Robust Visual Continual Learning with Multi-Prototype Supervision
Language-guided supervision, which utilizes a frozen semantic target from a Pretrained Language Model (PLM), has emerged as a promising paradigm for visual Continual Learning (CL). However, relying on a single target int…
Continual Learning