DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning
Document image segmentation is crucial in document analysis and recognition but remains challenging due to the heterogeneity of document formats and diverse segmentation tasks. Existing methods often treat these tasks separately, leading to limited generalization and resource wastage.This paper introduces DocSAM, a transformer-based unified framework for various document image segmentation tasks, including document layout analysis, multi-granularity text segmentation, and table structure recognition by modelling these tasks as a combination of instance and semantic segmentation.Specifically, DocSAM uses a Sentence BERT to map category names from each dataset into semantic queries of the same dimension as instance queries. These queries interact through attention mechanisms and are cross-attended with image features to predict instance and semantic segmentation masks. To predict instance categories, instance queries are dot-producted with semantic queries, and scores are normalized using softmax.As a result, DocSAM can be jointly trained on heterogeneous datasets, enhancing robustness and generalization while reducing computing and storage resources. Comprehensive evaluations show that DocSAM outperforms existing methods in accuracy, efficiency, and adaptability, highlighting its potential for advancing document image understanding and segmentation in various applications.
Code (0)
등록된 구현이 없습니다.
Tasks
Document Layout AnalysisImage SegmentationSegmentationSemantic SegmentationSentenceText SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentation
Instance-level segmentation of documents consists in assigning a class-aware and instance-aware label to each pixel of the image. It is a key step in document parsing for their understanding. In this paper, we present a …
DecoderInstance SegmentationSegmentationSemantic SegmentationMixed-Query Transformer: A Unified Image Segmentation Architecture
Existing unified image segmentation models either employ a unified architecture across multiple tasks but use separate weights tailored to each dataset, or apply a single set of weights to multiple datasets but are limit…
Data AugmentationImage SegmentationSegmentationSemantic SegmentationCMRAG: Co-modality-based visual document retrieval and question answering
Retrieval-Augmented Generation (RAG) has become a core paradigm in document question answering tasks. However, existing methods have limitations when dealing with multimodal documents: one category of methods relies on l…
Question AnsweringQuery by String word spotting based on character bi-gram indexing
In this paper we propose a segmentation-free query by string word spotting method. Both the documents and query strings are encoded using a recently proposed word representa- tion that projects images and strings into a …
AttributeRe-RankingRetrievalSegmentationDEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval
Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expans…