LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation

2025-03-25 · Han Chen, Zicong Jiang, Zining Zhang, Bingsheng He, Pingyi Luo, Mian Lu, Yuqiang Chen

We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving superior performance. Previous methods either assume that later tokens are more important or attempt to predict important tokens based on earlier attention patterns. Both approaches, however, can result in performance bottlenecks or frequent mispredictions. LogQuant takes a different approach. By applying a log-based filtering mechanism, it selectively compresses the KV Cache across the entire context, achieving better performance with the same or even reduced memory footprint compared to existing methods. In benchmark tests, it enhances throughput by 25% and boosts batch size by 60% without increasing memory consumption. For challenging tasks such as Math and Code Completion, LogQuant improves accuracy by 40% to 200% at the same compression ratio, outperforming comparable techniques.LogQuant integrates effortlessly with popular inference frameworks like Python's transformers library. Implementation can be available in https://github.com/Concyclics/LogQuantKV.

📄 PDF Abstract BibTeX arXiv:2503.19950

Code (1)

concyclics/logquantkv 공식 구현 pytorch

Tasks

Code CompletionLanguage ModelingLanguage ModellingLarge Language ModelMathQuantization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Accurate KV Cache Quantization with Outlier Tokens Tracing

2025-05-16 · Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li 외

The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also intro…

Quantization

Palu: Compressing KV-Cache with Low-Rank Projection

2024-07-30 · Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen 외

Post-training KV-Cache compression methods typically either sample a subset of effectual tokens or quantize the data into lower numerical bit width. However, these methods cannot exploit redundancy in the hidden dimensio…

GPU Quantization

XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression

2025-10-13 · Haoqi Yang, Yao Yao, Zuchao Li, Baoyuan Qi 외 arxiv

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text …

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

2024-05-23 · Yefei He, Luoming Zhang, Weijia Wu, Jing Liu 외

KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of t…

GPU GSM8K Quantization

SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

2024-05-10 · Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan 외

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substanti…