Docopilot: Improving Multimodal Models for Document-Level Understanding
Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Docopilot: Improving Multimodal Models for Document-Level Understanding
Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level da…
document understandingRAGRetrievalRetrieval-augmented GenerationDocMSU: A Comprehensive Benchmark for Document-level Multimodal Sarcasm Understanding
Multimodal Sarcasm Understanding (MSU) has a wide range of applications in the news field such as public opinion analysis and forgery detection. However, existing MSU benchmarks and approaches usually focus on sentence-l…
Object DetectionSarcasm DetectionSentenceMultimodal Graph RAG for Long-range Visually Rich Document Understanding
Multimodal large language models (MLLMs) are widely applied to visual document understanding. However, comprehending long documents remains an issue by the limited context window. Though recent multimodal retrieval-augme…
Visual Question Answering3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding
This paper presents a groundbreaking multimodal, multi-task, multi-teacher joint-grained knowledge distillation model for visually-rich form document understanding. The model is designed to leverage insights from both fi…
document understandingFormKnowledge DistillationToken-level Correlation-guided Compression for Efficient Multimodal Document Understanding
Cropping high-resolution document images into multiple sub-images is the most widely used approach for current Multimodal Large Language Models (MLLMs) to do document understanding. Most of current document understanding…
document understandingInformativeness