홈 › Papers

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

2024-08-07 · Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, dianhai yu, Hua Wu

Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosting these models is cost-prohibitive mainly due to the extensive memory consumption of KV Cache involving long-context modeling. Despite several works proposing to evict unnecessary tokens from the KV Cache, most of them rely on the biased local statistics of accumulated attention scores and report performance using unconvincing metric like perplexity on inadequate short-text evaluation. In this paper, we propose NACL, a general framework for long-context KV cache eviction that achieves more optimal and efficient eviction in a single operation during the encoding phase. Due to NACL's efficiency, we combine more accurate attention score statistics in PROXY TOKENS EVICTION with the diversified random eviction strategy of RANDOM EVICTION, aiming to alleviate the issue of attention bias and enhance the robustness in maintaining pivotal tokens for long-context modeling tasks. Notably, our method significantly improves the performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 50% with over 95% performance maintenance. The code is available at https://github.com/PaddlePaddle/Research/tree/master/NLP/ACL2024-NACL.

📄 PDF Abstract BibTeX arXiv:2408.03675

Code (2)

PaddlePaddle/Research 공식 구현 paddle

PaddlePaddle/Research/tree/master/NLP/ACL2024-NACL 공식 구현 paddle

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

2025-05-24 · YiXuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu 외

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challen…

Value-Aware Stochastic KV Cache Eviction for Reasoning Models

2026-06-02 · Ting-Yun Chang, Harvey Yiyun Fu, Deqing Fu, Chenghao Yang 외 arxiv

Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from…

In-context KV-Cache Eviction for LLMs via Attention-Gate

2024-10-15 · Zihao Zeng, Bokai Lin, Tianqi Hou, Hao Zhang 외

The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system. This paper enables a novel…

RTE

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

2026-02-03 · Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen 외 arxiv

Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant…

Reinforcement Learning

EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving

2025-12-16 · Shaoting Feng, Yuhan Liu, Hanchen Li, Xiaokun Chen 외 arxiv

Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity, so prior work has proposed to either e…