ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
Transformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes. In this paper, we propose ERNIE-Doc, a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism, enable ERNIE-Doc, which has a much longer effective context length, to capture the contextual information of a complete document. We pretrain ERNIE-Doc to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various experiments were conducted on both English and Chinese document-level tasks. ERNIE-Doc improved the state-of-the-art language modeling result of perplexity to 16.8 on WikiText-103. Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks, such as text classification and question answering.
Code (2)
Tasks
Language ModelingLanguage ModellingQuestion AnsweringSentiment AnalysisText ClassificationSimilar Papers 제목 키워드 기반
ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
Sparse Transformer has recently attracted a lot of attention since the ability for reducing the quadratic dependency on the sequence length. We argue that two factors, information bottleneck sensitivity and inconsistency…
Sparse Learningtext-classificationText ClassificationERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
Sparse Transformer has recently attracted a lot of attention since the ability for reducing the quadratic dependency on the sequence length. We argue that two factors, information bottleneck sensitivity and inconsistency…
Sparse Learningtext-classificationText ClassificationERNIE-Layout: Layout-Knowledge Enhanced Multi-modal Pre-training for Document Understanding
We propose ERNIE-Layout, a knowledge enhanced pre-training approach for visual document understanding, which incorporates layout-knowledge into the pre-training of visual document understanding to learn a better joint mu…
cross-modal alignmentDocument Classificationdocument understandingQuestion AnsweringERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding
Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge,…
document-image-classificationDocument Image Classificationdocument understandingimage-classification+6ERNIE-SPARSE: Robust Efficient Transformer Through Hierarchically Unifying Isolated Information
Sparse Transformer has recently attracted a lot of attention since the ability for reducing the quadratic dependency on the sequence length. In this paper, we argue that two factors could affect the robustness and causin…
Sensitivitytext-classificationText Classification