Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.
Code (0)
등록된 구현이 없습니다.
Tasks
AllMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Selective Attention Improves Transformer
Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selecti…
Language ModelingLanguage ModellingSparQ Attention: Bandwidth-Efficient LLM Inference
The computational difficulties of large language model (LLM) inference remain a significant obstacle to their widespread deployment. The need for many applications to support long input sequences and process them in larg…
Language ModelingLanguage ModellingLarge Language ModelDifferentially Private Attention Computation
Large language models (LLMs), especially those based on the Transformer architecture, have had a profound impact on various aspects of daily life, such as natural language processing, content generation, research methodo…
TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
Large language model (LLM) inference demands significant amount of computation and memory, especially in the key attention mechanism. While techniques, such as quantization and acceleration algorithms, like FlashAttentio…
Computational EfficiencyLanguage ModelingLanguage ModellingLarge Language Model+1Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
In this paper, we propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs), termed Efficient Attention Skipping (EAS). Concretely, we first reveal that multi-head at…
AllTransfer Learning