Revisiting Linformer with a modified self-attention with linear complexity
Although Transformer models such as Google's BERT and OpenAI's GPT-3 are successful in many natural language processing tasks, training and deploying these models are costly and inefficient.Even if pre-trained models are used, deploying these models still remained a challenge due to their large size. Apart from deployment, these models take higher time during inference restricting user-friendliness. The main bottleneck is self-attention which uses quadratic time and space with respect to the sequence length. In order to reduce the quadratic time complexity of the self-attention mechanism, Linformer by Facebook's AI research team was introduced where they showed that the self-attention mechanism can be approximated by a low-rank matrix and exploiting this finding, a new method for self-attention with linear time and space complexity was proposed by them. In the Linformer, the time complexity depends on the projection mapping dimension which acts as a hyperparameter and affects the performance of the model, tuning this hyperparameter can be time-consuming. In this paper, I proposed an alternative method for self-attention with linear complexity in time and space and is independent of the projection mapping dimension. Since this method works for long sequences this can be used for images as well as audios.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rethinking Transformer Connectivity: TLinFormer, A Path to Exact, Full Context-Aware Linear Attention
The Transformer architecture has become a cornerstone of modern artificial intelligence, but its core self-attention mechanism suffers from a complexity bottleneck that scales quadratically with sequence length, severely…
Linformer: Self-Attention with Linear Complexity
Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly f…
Language ModellingStateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation
Effective navigation intelligence relies on long-term memory to support both immediate generalization and sustained adaptation. However, existing approaches face a dilemma: modular systems rely on explicit mapping but la…
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences
Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. To address this limitation, Linformer and Informer are proposed to red…
Spatially Aware Linear Transformer (SAL-T) for Particle Jet Tagging
Transformers are very effective in capturing both global and local correlations within high-energy particle collisions, but they present deployment challenges in high-data-throughput environments, such as the CERN LHC. T…
Point Cloud ClassificationJet Tagging