DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs

2024-12-19 · Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, Liang Ding

Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released.

📄 PDF Abstract BibTeX arXiv:2412.14838

Code (0)

등록된 구현이 없습니다.

Tasks

RAG

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight Decay 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

WordPiece 설명 없음

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

2026-06-16 · Ning Ni, Yingjie Lao arxiv

Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.…

Transactional Attention: Semantic Sponsorship for KV-Cache Retention

2026-04-13 · Abhinaba Basu arxiv

At K=16 tokens (0.4% of a 4K context), every existing KV-cache compression method achieves 0% on credential retrieval. The failure mode is dormant tokens: credentials, API keys, and configuration values that receive near…

VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

2024-10-29 · Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, Panpan Xu

Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visua…

GPU Language Modeling Language Modelling

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

2025-10-26 · Jinhan Chen, Jianchun Liu, Hongli Xu, Xianjun Gao 외 arxiv

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discar…

Semantic Segmentation

Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach

2025-11-20 · Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu 외 arxiv

Multimodal large language models suffer from substantial inference overhead since multimodal KV Cache grows proportionally with the visual input length. Existing multimodal KV Cache compression methods mostly rely on att…