Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval
Zero-shot learning offers an efficient solution for a machine learning model to treat unseen categories, avoiding exhaustive data collection. Zero-shot Sketch-based Image Retrieval (ZS-SBIR) simulates real-world scenarios where it is hard and costly to collect paired sketch-photo samples. We propose a novel framework that indirectly aligns sketches and photos by contrasting them through texts, removing the necessity of access to sketch-photo pairs. With an explicit modality encoding learned from data, our approach disentangles modality-agnostic semantics from modality-specific information, bridging the modality gap and enabling effective cross-modal content retrieval within a joint latent space. From comprehensive experiments, we verify the efficacy of the proposed model on ZS-SBIR, and it can be also applied to generalized and fine-grained settings.
Code (1)
Tasks
Image RetrievalRepresentation LearningRetrievalSketch-Based Image RetrievalZero-Shot LearningSimilar Papers 제목 키워드 기반
Ontology-aware Network for Zero-shot Sketch-based Image Retrieval
Zero-Shot Sketch-Based Image Retrieval (ZSSBIR) is an emerging task. The pioneering work focused on the modal gap but ignored inter-class information. Although recent work has begun to consider the triplet-based or contr…
Image RetrievalRetrievalSketch-Based Image RetrievalTripletModality-Aware Triplet Hard Mining for Zero-shot Sketch-Based Image Retrieval
This paper tackles the Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) problem from the viewpoint of cross-modality metric learning. This task has two characteristics: 1) the zero-shot setting requires a metric space wi…
Image RetrievalMetric LearningRetrievalSketch-Based Image Retrieval+1Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval
This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR), which aims to use sketches from unseen categories as queries to match the images of the same category. Due to the large cross-modality d…
Image RetrievalKnowledge DistillationRetrievalSketch-Based Image RetrievalA Zero-Shot Sketch-based Inter-Modal Object Retrieval Scheme for Remote Sensing Images
Conventional existing retrieval methods in remote sensing (RS) are often based on a uni-modal data retrieval framework. In this work, we propose a novel inter-modal triplet-based zero-shot retrieval scheme utilizing a sk…
RetrievalTripletOn Learning Semantic Representations for Million-Scale Free-Hand Sketches
In this paper, we study learning semantic representations for million-scale free-hand sketches. This is highly challenging due to the domain-unique traits of sketches, e.g., diverse, sparse, abstract, noisy. We propose a…
Deep HashingLearning Semantic RepresentationsRetrievalZero-Shot Learning