paper-with-me

홈 › Papers

Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices

2024-10-02 · Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, Zhiyuan Liu

Scaling the input context length of a large language model (LLM) incurs a significant increase in computation cost and memory footprint to maintain the attention key-value (KV) cache. Existing KV cache compression methods suffer from inefficient compression strategies and limited memory reduction effects, making it difficult for LLMs to conduct long-context inference on consumer-grade devices, especially when inferring long-context stream input. Such obstacles prevent consumer-grade devices from supporting more complex applications, creating challenges for the democratization of LLMs. To overcome this, we propose Locret, the first framework to create an eviction policy compatible with chunked prefill. By evaluating the causal importance of KV cache units by learnable retaining heads, Locret enables precise eviction of cache units, facilitating efficient long-context inference. In our extensive empirical studies, Locret outperforms the recent popular and competitive approaches in terms of memory efficiency and generation quality -- Locret achieves up to 20x of KV cache compression ratio within less than 10% performance loss. Furthermore, Locret achieves 128K+ long-context inference on a single NVIDIA 4090 GPU without compromising generation quality and only costs <1 GPU hour of additional training.

📄 PDF Abstract BibTeX arXiv:2410.01805

Code (1)

huangyuxiang03/Locret 공식 구현 jax

Tasks

GPULanguage ModelingLanguage ModellingLarge Language ModelQuantization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Attention 설명 없음
SFT Shrink and Fine-Tune, or SFT, is a type of distillation that avoids explicit distillation by copying parameters to a student student model and then fine-tuning.…

Similar Papers 제목 키워드 기반

Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

2026-05-10 · Ngoc Bui, Hieu Trung Nguyen, Arman Cohan, Rex Ying arxiv

The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative t…

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

2026-05-25 · Xintong Yang, Hao Gu, Binxing Xu, Lujun Li 외 arxiv

Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long co…

LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference

2025-03-11 · Guangtao Wang, Shubhangi Upasani, Chen Wu, Darshan Gandhi 외

Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and the high computational complexity of att…

MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

2025-06-06 · Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode Wang 외

This paper introduces MadaKV, a modality-adaptive key-value (KV) cache eviction strategy designed to enhance the efficiency of multimodal large language models (MLLMs) in long-context inference. In multimodal scenarios, …

OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference

2025-10-09 · Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, Enmao Diao arxiv

Large language models (LLMs) with extended context windows enable powerful applications but impose significant memory overhead, as caching all key-value (KV) states scales linearly with sequence length and batch size. Ex…