KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

2024-10-24 · Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, Zhi Chen

The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \textit{KVSharer} can reduce KV cache computation by 30\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.

📄 PDF Abstract BibTeX arXiv:2410.18517

Code (1)

yangyifei729/kvsharer 공식 구현 pytorch

Tasks

GPU

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

2024-04-07 · ZiHao Wang, Bin Cui, Shaoduo Gan

Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence …

Language Modelling Large Language Model Management

KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

2025-02-06 · Xing Li, Zeyu Xing, Yiming Li, Linping Qu 외

KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three uns…

Mathematical Reasoning Quantization Sensitivity

MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference

2025-02-24 · Zhongwei Wan, Hui Shen, Xin Wang, Che Liu 외

Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths,…

LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference

2025-12-18 · Harsh Vardhan Bansal arxiv

Transformer-based language models have achieved remarkable performance across a wide range of tasks, yet their high inference latency poses a significant challenge for real-timeand large-scale deployment. While existing …

Semantic Similarity

Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference

2026-01-12 · Rei Taniguchi, Yuyang Dong, Makoto Onizuka, Chuan Xiao arxiv

Due to the prevalence of large language models (LLMs), key-value (KV) cache reduction for LLM inference has received remarkable attention. Among numerous works that have been proposed in recent years, layer-wise token pr…