Transformer tricks: Precomputing the first layer
This micro-paper describes a trick to speed up inference of transformers with RoPE (such as LLaMA, Mistral, PaLM, and Gemma). For these models, a large portion of the first transformer layer can be precomputed, which results in slightly lower latency and lower cost-per-token. Because this trick optimizes only one layer, the relative savings depend on the total number of layers. For example, the maximum savings for a model with only 4 layers (such as Whisper tiny) is limited to 25%, while a 32-layer model is limited to 3% savings. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations
Deep pretrained transformer networks are effective at various ranking tasks, such as question answering and ad-hoc document ranking. However, their computational expenses deem them cost-prohibitive in practice. Our propo…
Document RankingQuestion AnsweringRe-RankingTransformer tricks: Removing weights for skipless transformers
He and Hofmann (arXiv:2311.01906) detailed a skipless transformer without the V and P (post-attention projection) linear layers, which reduces the total number of weights. However, this scheme is only applicable to MHA (…
LazyFormer: Self Attention with Lazy Update
Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the self-attention module, which is computationally expensive. In this paper, we propose a simple but effect…
Improving Transformer Optimization Through Better Initialization
The Transformer architecture has achieved considerable success in areas such as language modeling and machine translation. The key component of the Transformer is the attention layer that enables the model to focus on im…
DecoderLanguage ModelingLanguage ModellingMachine Translation+1Improving Transformer Optimization Through Better Initialization
The Transformer architecture has achieved considerable success in areas such as language modeling and machine translation. The key component of the Transformer is the attention layer that enables the model to focus on im…
DecoderLanguage ModelingLanguage ModellingMachine Translation+1