Composing Text and Image for Image Retrieval - An Empirical Odyssey
In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar but are modified in small ways, such as being taken at nighttime instead of during the day. To tackle this task, we learn a similarity metric between a target image and a source image plus source text, an embedding and composing function such that target image feature is close to the source image plus text composition feature. We propose a new way to combine image and text using such function that is designed for the retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to classify input queries, in addition to image retrieval.
Code (4)
Tasks
Image RetrievalImage Retrieval with Multi-Modal QueryRetrievalSimilar Papers 제목 키워드 기반
Learning Joint Visual Semantic Matching Embeddings for Language-guided Retrieval
Interactive image retrieval is an emerging research topic with the objective of integrating inputs from multiple modalities as query for retrieval, e.g., textual feedback from users to guide, modify or refine image retri…
Image RetrievalRetrievalSpecificityText RetrievalMIRAGE: Runtime Scheduling for Multi-Vector Image Retrieval with Hierarchical Decomposition
To effectively leverage user-specific data, retrieval augmented generation (RAG) is employed in multimodal large language model (MLLM) applications. However, conventional retrieval approaches often suffer from limited re…
Image RetrievalProbabilistic Compositional Embeddings for Multimodal Image Retrieval
Existing works in image retrieval often consider retrieving images with one or two query inputs, which do not generalize to multiple queries. In this work, we investigate a more challenging scenario for composing multipl…
Image RetrievalRetrievalFrom Mapping to Composing: A Two-Stage Framework for Zero-shot Composed Image Retrieval
Composed Image Retrieval (CIR) is a challenging multimodal task that retrieves a target image based on a reference image and accompanying modification text. Due to the high cost of annotating CIR triplet datasets, zero-s…
Image RetrievalRetrievalTripletProgressive Learning for Image Retrieval with Hybrid-Modality Queries
Image retrieval with hybrid-modality queries, also known as composing text and image for image retrieval (CTI-IR), is a retrieval task where the search intention is expressed in a more complex query format, involving bot…
Image RetrievalImage-text RetrievalRetrievalText Retrieval