Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

2021-02-18 · Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, Gabriela Pałka

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.

📄 PDF Abstract BibTeX arXiv:2102.09550

Code (1)

uakarsh/TiLT-Implementation pytorch

Tasks

Decoder Document Image Classificationdocument understandingVisual Question Answering Visual Question Answering (VQA)

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Arctic-TILT. Business Document Understanding at Sub-Billion Scale

2024-08-08 · Łukasz Borchmann, Michał Pietruszka, Wojciech Jaśkowski, Dawid Jurkiewicz 외

The vast portion of workloads employing LLMs involves answering questions grounded on PDF or scan content. We introduce the Arctic-TILT achieving accuracy on par with models 1000$\times$ its size on these use cases. It c…

document understanding GPU

At FullTilt: Real-Time Open-Set 3D Macromolecule Detection Directly from Tilted 2D Projections

2026-04-12 · Ming-Yang Ho, Alberto Bartesaghi arxiv

Open-set 3D macromolecule detection in cryogenic electron tomography eliminates the need for target-specific model retraining. However, strict VRAM constraints prohibit processing an entire 3D tomogram, forcing current m…

Scalable Psychological Momentum Forecasting in Esports

2020-01-30 · Alfonso White, Daniela M. Romano

The world of competitive Esports and video gaming has seen and continues to experience steady growth in popularity and complexity. Correspondingly, more research on the topic is being published, ranging from social netwo…

Benchmarking

Tilt-Ropter: A Fully Actuated Hybrid Aerial-Terrestrial Vehicle with Tilt Rotors and Passive Wheels

2026-02-02 · Ruoyu Wang, Xuchen Liu, Zongzhou Wu, Zixuan Guo 외 arxiv

In this work, we present Tilt-Ropter, a fully actuated hybrid aerial-terrestrial vehicle (HATV) that integrates tilt rotors with passive wheels to enable efficient multi-modal locomotion. Unlike conventional underactuate…

RotorSuite: A MATLAB/Simulink Toolbox for Tilt Multi-Rotor UAV Modeling

2026-02-21 · Nicola Cigarini, Giulia Michieletto, Angelo Cenedese arxiv

In recent years, aerial platforms have evolved from passive flying sensors into versatile, contact-aware robotic systems, leading to rapid advances in platform design. Standard coplanar and collinear quadrotors have been…