ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

2025-02-01 · Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, Xiaowen Chu

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem.

📄 PDF Abstract BibTeX arXiv:2502.00299

Code (0)

등록된 구현이 없습니다.

Tasks

GPUGSM8KIn-Context Learning

Methods 이 논문이 사용한 방법론

Focus 설명 없음

Value-Guided KV Compression for LLMs via Approximated CUR Decomposition

2025-09-18 · Ayan Sengupta, Siddhant Chaudhary, Tanmoy Chakraborty arxiv

Key-value (KV) cache compression has emerged as a critical technique for reducing the memory and latency overhead of autoregressive language models during inference. Prior approaches predominantly rely on query-key atten…

DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity

2026-02-08 · Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang 외 arxiv

The deployment of efficient long-context LLMs in applications like autonomous agents, long-chain reasoning, and creative writing is fundamentally bottlenecked by the linear growth of KV cache memory. Existing compression…

GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs

2026-05-29 · Junjie Peng, You Wu, Haoyi Wu, Jialong Han 외 arxiv

Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substantial memory overhead, motivating KV-cach…

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference

2025-03-23 · Youhui Zuo, Sibo Wei, Chen Zhang, Zhuorui Liu 외

With the advancements in long-context inference capabilities of large language models (LLMs), the KV cache has become one of the foundational components. However, its substantial GPU memory consumption makes KV cache com…

GPU

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

2026-05-28 · Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir Singh arxiv

Key-Value (KV) cache remains a major bottleneck for deploying Large Language Models (LLMs) in long-generation tasks. Prior work often applies uniform compression across both prefill and decoding caches, but compressing t…