MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin

2023-10-21 · Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, Ge Yu

This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVEL encodes queries and multi-modal documents with a unified encoder model, which helps to alleviate the modality gap between images and texts. Specifically, we enable the image understanding ability of the well-trained dense retriever, T5-ANCE, by incorporating the visual module's encoded image features as its inputs. To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and extracts the related text and image documents from anchor-linked web pages. Our experiments show that MARVEL significantly outperforms the state-of-the-art methods on the multi-modal retrieval dataset WebQA and ClueWeb22-MM. MARVEL provides an opportunity to broaden the advantages of text retrieval to the multi-modal scenario. Besides, we also illustrate that the language model has the ability to extract image semantics and partly map the image features to the input word embedding space. All codes are available at https://github.com/OpenMatch/MARVEL.

📄 PDF Abstract BibTeX arXiv:2310.14037

Code (1)

openmatch/marvel 공식 구현 pytorch

Tasks

Language Modelling Retrieval Text Retrieval

MARVEL: Universal Murray's Law-informed Vessel Tree Segmentation and Topology Estimation

2026-05-25 · Yi Zhou, Thiara Sana Ahmed, Jacqueline Chua, Meng Wang 외 arxiv

Vascular circulation follows fundamental biophysical principles that optimize mass transport and metabolic energy expenditure, which can be effectively modeled by Murray's law. However, contemporary deep learning methods…

Unlocking Dense Metric Depth Estimation in VLMs

2026-05-15 · Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu 외 arxiv

Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visu…

Spatial Reasoning Depth Estimation

MARVEL: Multi-Agent Reinforcement-Learning for Large-Scale Variable Speed Limits

2023-10-18 · Yuhang Zhang, Marcos Quinones-Grueiro, Zhiyao Zhang, Yanbing Wang 외

Variable Speed Limit (VSL) control acts as a promising highway traffic management strategy with worldwide deployment, which can enhance traffic safety by dynamically adjusting speed limits according to real-time traffic …

Decision Making Management Multi-agent Reinforcement Learning reinforcement-learning

MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

2026-04-23 · Juan Li, Chuanghao Ding, Xujie Zhang, Cam-Tu Nguyen arxiv

Universal Multimodal Retrieval (UMR) aims to map different modalities (e.g., visual and textual) into a shared embedding space for multi-modal retrieval. Existing UMR methods can be broadly divided into two categories: e…

U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs

2025-07-20 · Xiaojie Li, Chu Li, Shi-Zhe Chen, Xi Chen arxiv

Universal multimodal retrieval (UMR), which aims to address complex retrieval tasks where both queries and candidates span diverse modalities, has been significantly advanced by the emergence of MLLMs. While state-of-the…

Contrastive Learning Image Retrieval Video Retrieval