Lossless KV Cache Compression to 2%

2024-10-20 · Zhen Yang, J. N. Han, Kan Wu, Ruobing Xie, An Wang, Xingwu Sun, Zhanhui Kang

Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up inference, maintaining a key-value (KV) cache memory is essential. Nonetheless, the growing demands for KV cache memory create significant hurdles for efficient implementation. This work introduces a novel architecture, Cross-Layer Latent Attention (CLLA), aimed at compressing the KV cache to less than 2% of its original size while maintaining comparable performance levels. CLLA integrates multiple aspects of KV cache compression, including attention head/dimension reduction, layer sharing, and quantization techniques, into a cohesive framework. Our extensive experiments demonstrate that CLLA achieves lossless performance on most tasks while utilizing minimal KV cache, marking a significant advancement in practical KV cache compression.

📄 PDF Abstract BibTeX arXiv:2410.15252

Code (0)

등록된 구현이 없습니다.

Tasks

Dimensionality Reduction Quantization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

LoMA: Lossless Compressed Memory Attention

2024-01-16 · Yumeng Wang, Zhenyang Xiao

Large Language Models (LLMs) face limitations due to the high demand on GPU memory and computational resources when handling long contexts. While sparsify the Key-Value (KV) cache of transformer model is a typical strate…

GPU

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

2026-05-03 · Yipin Guo, Siddharth Joshi arxiv

Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill w…

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM

2024-03-08 · Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong 외

Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to…

Quantization

A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs

2026-07-14 · Rahul Krishnan, Volker Schulz arxiv

The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the throughput ceili…

TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization

2026-03-29 · Dipkumar Patel arxiv

We compress KV cache entries by quantizing angles in the Fast Walsh-Hadamard domain, where a random diagonal rotation makes consecutive element pairs approximately uniformly distributed on the unit circle. We extend this…