Composed Multi-modal Retrieval: A Survey of Approaches and Applications
With the rapid growth of multi-modal data from social media, short video platforms, and e-commerce, content-based retrieval has become essential for efficiently searching and utilizing heterogeneous information. Over time, retrieval techniques have evolved from Unimodal Retrieval (UR) to Cross-modal Retrieval (CR) and, more recently, to Composed Multi-modal Retrieval (CMR). CMR enables users to retrieve images or videos by integrating a reference visual input with textual modifications, enhancing search flexibility and precision. This paper provides a comprehensive review of CMR, covering its fundamental challenges, technical advancements, and categorization into supervised, zero-shot, and semi-supervised learning paradigms. We discuss key research directions, including data augmentation, model architecture, and loss optimization in supervised CMR, as well as transformation frameworks and external knowledge integration in zero-shot CMR. Additionally, we highlight the application potential of CMR in composed image retrieval, video retrieval, and person retrieval, which have significant implications for e-commerce, online search, and public security. Given its ability to refine and personalize search experiences, CMR is poised to become a pivotal technology in next-generation retrieval systems. A curated list of related works and resources is available at: https://github.com/kkzhang95/Awesome-Composed-Multi-modal-Retrieval
Code (1)
Tasks
Cross-Modal RetrievalData AugmentationImage RetrievalPerson RetrievalRetrievalVideo RetrievalSimilar Papers 제목 키워드 기반
MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like…
Contrastive LearningA Comprehensive Survey on Composed Image Retrieval
Composed Image Retrieval (CIR) is an emerging yet challenging task that allows users to search for target images using a multimodal query, comprising a reference image and a modification text specifying the user's desire…
AttributeImage RetrievalRetrievalSurveyA Survey of Multimodal Composite Editing and Retrieval
In the real world, where information is abundant and diverse across different modalities, understanding and utilizing various data types to improve retrieval systems is a key focus of research. Multimodal composite retri…
RetrievalSurveyNEUCORE: Neural Concept Reasoning for Composed Image Retrieval
Composed image retrieval which combines a reference image and a text modifier to identify the desired target image is a challenging task, and requires the model to comprehend both vision and language modalities and their…
Concept AlignmentImage RetrievalMultiple Instance LearningRetrieval+1Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
Document understanding is critical for applications from financial analysis to scientific discovery. Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs),…
Document AI