MATrIX -- Modality-Aware Transformer for Information eXtraction
We present MATrIX - a Modality-Aware Transformer for Information eXtraction in the Visual Document Understanding (VDU) domain. VDU covers information extraction from visually rich documents such as forms, invoices, receipts, tables, graphs, presentations, or advertisements. In these, text semantics and visual information supplement each other to provide a global understanding of the document. MATrIX is pre-trained in an unsupervised way with specifically designed tasks that require the use of multi-modal information (spatial, visual, or textual). We consider the spatial and text modalities all at once in a single token set. To make the attention more flexible, we use a learned modality-aware relative bias in the attention mechanism to modulate the attention between the tokens of different modalities. We evaluate MATrIX on 3 different datasets each with strong baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
document understandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Point-aware Interaction and CNN-induced Refinement Network for RGB-D Salient Object Detection
By integrating complementary information from RGB image and depth map, the ability of salient object detection (SOD) for complex and challenging scenes can be improved. In recent years, the important role of Convolutiona…
object-detectionObject DetectionRGB-D Salient Object DetectionSalient Object DetectionNestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation
Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusi…
Brain Tumor SegmentationDecoderMRI segmentationSegmentation+1Cross-Modality Fusion Transformer for Multispectral Object Detection
Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effectiv…
Multispectral Object DetectionObjectobject-detectionObject Detection+1Modality-aware Transformer for Financial Time series Forecasting
Time series forecasting presents a significant challenge, particularly when its accuracy relies on external data sources rather than solely on historical values. This issue is prevalent in the financial sector, where the…
Feature ImportanceTime SeriesTime Series ForecastingMulti-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Clasification
Deep learning approaches, together with neuroimaging techniques, play an important role in psychiatric disorders classification. Previous studies on psychiatric disorders diagnosis mainly focus on using functional connec…
Functional ConnectivityTime Series