Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression

2026-03-21 · Ruijie Miao, Zhiming Wang, Wang Li, Shiwei Wu, Shufan Liu, Yanbing Jiang, Tong Yang arxiv

Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token eviction methods reduce memory by discarding less important tokens, which can be viewed as a coarse form of dimensionality reduction that assigns each token either zero or full dimension. We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level, and MixedDimKV-H, which further integrates head-level importance information. Experiments on long-context benchmarks show that MixedDimKV outperforms prior KV cache compression methods that do not rely on head-level importance profiling. When equipped with the same head-level importance information, MixedDimKV-H consistently outperforms HeadKV. Notably, our approach achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache. Furthermore, in the Needle-in-a-Haystack test, our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache.

📄 PDF Abstract BibTeX arXiv:2603.20616

Code (0)

등록된 구현이 없습니다.

Tasks

Dimensionality Reduction

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

2026-05-24 · Yubo Li, Yidi Miao arxiv

Long-horizon LLM inference turns the key--value (KV) cache into the dominant GPU memory consumer and makes per-token attention increasingly expensive. Many common eviction policies use static recency windows or historica…

Beyond Compaction: Structured Context Eviction for Long-Horizon Agents

2026-05-01 · Andrew Semenov, Svyatoslav Dorofeev arxiv

We present Context Window Lifecycle (CWL), a context-management scheme that gives long-horizon LLM agents an effectively unbounded working horizon. As a session accumulates history, CWL keeps the context within budget th…

Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets

2026-06-22 · Duc Duong, Hoang Anh Duy Le, Jianwen Xie, Anshumali Shrivastava 외 arxiv

Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream. Existing methods all share th…

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

2025-10-26 · Jinhan Chen, Jianchun Liu, Hongli Xu, Xianjun Gao 외 arxiv

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discar…

Semantic Segmentation

Learning to Evict from Key-Value Cache

2026-02-10 · Luca Moschella, Laura Manduchi, Ozan Sener arxiv

The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost…

Reinforcement Learning Passage Retrieval