Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

2026-05-03 · Nikita Agrawal, Ruben Mayer arxiv

Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.

📄 PDF Abstract BibTeX arXiv:2607.05399

Code (0)

등록된 구현이 없습니다.

Tasks

Few-Shot Learning

UltraQuant: 4-bit KV Caching for Context-Heavy Agents

2026-06-18 · Inesh Chakrabarti, David Limpus, Aditi Ghai Rana, Bowen Bao 외 arxiv

Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV…

Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence

2026-07-23 · Jay Gor, Karm Dave, Akshita Abrol, Rajesh Gupta 외 arxiv

Efficient multimodal inference is increasingly constrained not only by model quality or FLOP count, but also by the cost of preserving, moving, routing, caching, and quantizing multimodal representations under latency, m…

Evaluating Memento Service Optimizations

2019-05-31 · Martin Klein, Lyudmila Balakireva, Harihar Shankar

Services and applications based on the Memento Aggregator can suffer from slow response times due to the federated search across web archives performed by the Memento infrastructure. In an effort to decrease the response…

BIG-bench Machine Learning

OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism

2026-03-15 · Xiangyu Li, Huaizhi Tang, Xin Ding, Weijun Wang 외 arxiv

Embodied AI agents increasingly require parallel execution of multiple tasks, such as manipulation, conversation, and memory construction, from shared observations under distinct time constraints. Recent Mixture-of-Trans…

HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming

2025-12-24 · Haonan Qiu, Shikun Liu, Zijian Zhou, Zhaochong An 외 arxiv

High-resolution video generation, while crucial for digital media and film, is computationally bottlenecked by the quadratic complexity of diffusion models, making practical inference infeasible. To address this, we intr…

Video Generation