홈 › Papers

CommVQ: Commutative Vector Quantization for KV Cache Compression

2025-06-23 · Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, Chuang Gan

Large Language Models (LLMs) are increasingly used in applications requiring long context lengths, but the key-value (KV) cache often becomes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference. We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Position Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: https://github.com/UMass-Embodied-AGI/CommVQ.

📄 PDF Abstract BibTeX arXiv:2506.18879

Code (1)

umass-embodied-agi/commvq 공식 구현 pytorch

Tasks

GPUGSM8K Quantization

Residual vector quantization for KV cache compression in large language model

2024-10-21 · Ankur Kumar

KV cache compression methods have mainly relied on scalar quantization techniques to reduce the memory requirements during decoding. In this work, we apply residual vector quantization, which has been widely used for hig…

Audio Compression Language Modeling Language Modelling Large Language Model+1

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

2026-03-17 · Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu 외 arxiv

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely o…

QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering

2024-07-04 · Yanshu Wang, Wang Li, Zhaoqian Yao, Tong Yang

The matrix quantization entails representing matrix elements in a more space-efficient form to reduce storage usage, with dequantization restoring the original matrix for use. We formulate the Quantization Error Minimiza…

Computational Efficiency Edge-computing Image Compression Large Language Model+4

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

2025-01-31 · Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev 외

Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value cach…

GPU Quantization

GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models

2026-01-08 · Maanas Taneja, Purab Shingvi arxiv

The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We …