MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents

2025-01-15 · Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, Yong liu

Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of a comprehensive and robust benchmark to effectively evaluate the performance of systems in such tasks. To address this gap, this work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval. The former evaluates the performance of identifying the most relevant pages within a long document, while the later assesses the ability of detecting specific layouts, providing a more fine-grained measure than whole-page analysis. A layout refers to a variety of elements, including textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels, making it a valuable resource in multimodal document retrieval for both training and evaluation. Through rigorous experiments, we demonstrate that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR training set effectively enhances the performance of multimodal document retrieval and (iii) text retrievers leveraging VLM-text significantly outperforms retrievers relying on OCR-text. Our dataset is available at https://mmdocrag.github.io/MMDocIR/.

📄 PDF Abstract BibTeX arXiv:2501.08828

Code (0)

등록된 구현이 없습니다.

Tasks

Benchmarking Optical Character Recognition (OCR)Retrieval

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

2026-05-26 · Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao, Yiming Du 외 arxiv

Large vision-language models increasingly rely on long-context modeling to reason over documents, hour-level videos, and long-horizon agent trajectories, requiring them to locate relevant evidence across interleaved text…

Image Retrieval Head Detection Text Retrieval

Very Efficient Listwise Multimodal Reranking for Long Documents

2026-05-12 · Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh arxiv

Listwise reranking is a key yet computationally expensive component in vision-centric retrieval and multimodal retrieval-augmented generation (M-RAG) over long documents. While recent VLM-based rerankers achieve strong a…

Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)

2026-06-02 · Jingbiao Mei arxiv

Retrieval over visually-rich documents, pages that interleave text with figures, tables, and charts, is essential for multimodal retrieval-augmented generation, yet most retrievers still discard the visual channel. The \…

A Multi-Granularity Retrieval Framework for Visually-Rich Documents

2025-05-01 · Mingjun Xu, Zehui Wang, Hengxing Cai, Renxin Zhong

Retrieval-augmented generation (RAG) systems have predominantly focused on text-based retrieval, limiting their effectiveness in handling visually-rich documents that encompass text, images, tables, and charts. To bridge…

Language Modeling Language Modelling RAG Retrieval+1

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

2025-06-14 · Mingjun Xu, Jinhan Dong, Jue Hou, Zehui Wang 외

Multimodal document retrieval systems enable information access across text, images, and layouts, benefiting various domains like document-based question answering, report analysis, and interactive content summarization.…

Instruction Following Multimodal Reasoning Question Answering Reinforcement Learning (RL)+2