H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
We describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hierarchical Matrix (H-Matrix) developed by the numerical analysis community, and has linear run time and memory complexity. We perform extensive experiments to show that the inductive bias embodied by our hierarchical attention is effective in capturing the hierarchical structure in the sequences typical for natural language and vision tasks. Our method is superior to alternative sub-quadratic proposals by over +6 points on average on the Long Range Arena benchmark. It also sets a new SOTA test perplexity on One-Billion Word dataset with 5x fewer model parameters than that of the previous-best Transformer-based models.
Code (2)
Tasks
Inductive BiasLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast Multipole Attention: A Divide-and-Conquer Attention Mechanism for Long Sequences
Transformer-based models have achieved state-of-the-art performance in many areas. However, the quadratic complexity of self-attention with respect to the input length hinders the applicability of Transformer-based model…
Language ModelingLanguage ModellingDissociating Decodability and Causal Use in Bracket-Sequence Transformers
When trained on tasks requiring an understanding of hierarchical structure, transformers have been found to represent this hierarchy in distinct ways: in the geometry of the residual stream, and in stack-like attention p…
STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition
We study the problem of human action recognition using motion capture (MoCap) sequences. Unlike existing techniques that take multiple manual steps to derive standardized skeleton representations as model input, we propo…
Action RecognitionTemporal Action LocalizationHierarchical Learning for Generation with Long Source Sequences
One of the challenges for current sequence to sequence (seq2seq) models is processing long sequences, such as those in summarization and document level machine translation tasks. These tasks require the model to reason a…
DecoderDocument Level Machine TranslationDocument SummarizationDocument Translation+6Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers
We present a novel bi-directional Transformer architecture (BiXT) which scales linearly with input size in terms of computational cost and memory consumption, but does not suffer the drop in performance or limitation to …
image-classificationImage ClassificationImage SegmentationRetrieval+2