Linear attention is (maybe) all you need (to understand transformer optimization)
Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization.
Code (1)
Tasks
AllMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attention
Recently, Transformers have shown promising performance in various vision tasks. To reduce the quadratic computation complexity caused by each query attending to all keys/values, various methods have constrained the rang…
image-classificationImage ClassificationInstance Segmentationobject-detection+2B-Cos Aligned Transformers Learn Human-Interpretable Features
Vision Transformers (ViTs) and Swin Transformers (Swin) are currently state-of-the-art in computational pathology. However, domain experts are still reluctant to use these models due to their lack of interpretability. Th…
Pay Attention to Virality: understanding popularity of social media videos with the attention mechanism
Predicting popularity of social media videos before they are published is a challenging task, mainly due to the complexity of content distribution network as well as the number of factors that play part in this process. …
Fastformer: Additive Attention Can Be All You Need
Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still eit…
AllNews RecommendationText ClassificationText SummarizationMomentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization
Transformers have achieved remarkable success in sequence modeling and beyond but suffer from quadratic computational and memory complexities with respect to the length of the input sequence. Leveraging techniques includ…
Image GenerationMachine Translation