NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

2025-05-22 · Zhihang Cai, Xingjun Zhang, Zhendong Tan, Zheng Wei

Large Language Models (LLMs) have demonstrated remarkable proficiency across a wide range of tasks. However, LLMs often require larger batch sizes to enhance throughput or longer context lengths to meet task demands, which significantly increases the memory resource consumption of the Key-Value (KV) cache during inference, becoming a major bottleneck in LLM deployment. To address this issue, quantization is a common and straightforward approach. Currently, quantization methods for activations are limited to 8-bit, and quantization to even lower bits can lead to substantial accuracy drops. To further save space by quantizing the KV cache to even lower bits, we analyzed the element distribution of the KV cache and designed the NQKV algorithm. Since the elements within each block of the KV cache follow a normal distribution, NQKV employs per-block quantile quantization to achieve information-theoretically optimal quantization error. Without significantly compromising model output quality, NQKV enables the OPT model to perform inference with an 2x larger batch size or a 4x longer context length, and it improves throughput by 9.3x compared to when the KV cache is not used.

📄 PDF Abstract BibTeX arXiv:2505.16210

Code (0)

등록된 구현이 없습니다.

Tasks

Quantization

Methods 이 논문이 사용한 방법론

OPT OPT is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. The model uses an AdamW optimizer and weight decay of 0.1. It follows a linear…

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

2025-05-23 · Donghyun Son, Euntae Choi, Sungjoo Yoo

Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently ado…

Language Modeling Language Modelling Large Language Model Quantization

InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

2026-02-26 · Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross arxiv

When transformer-based language models are deployed for text generation, most of the inference time is spent in the decoding stage, where output tokens are generated sequentially. Reducing the hardware cost of each decod…

Text Generation

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

2024-05-23 · Yefei He, Luoming Zhang, Weijia Wu, Jing Liu 외

KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of t…

GPU GSM8K Quantization

PatternKV: Flattening KV Representation Expands Quantization Headroom

2025-10-05 · Ji Zhang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan 외 arxiv

KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a …

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

2026-06-02 · Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang 외 arxiv

Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows. KV-cache quantization can help improve…