Transactional Attention: Semantic Sponsorship for KV-Cache Retention

2026-04-13 · Abhinaba Basu arxiv

At K=16 tokens (0.4% of a 4K context), every existing KV-cache compression method achieves 0% on credential retrieval. The failure mode is dormant tokens: credentials, API keys, and configuration values that receive near-zero attention but become essential at generation time. Because these tokens lack the statistical signals that eviction policies rely on, no method based on attention scores, reconstruction loss, or learned retention gates retains them. We introduce Transactional Attention (TA), a sponsorship mechanism in which structural anchor patterns (e.g., "key:", "password:") protect adjacent value-bearing tokens from eviction. TA achieves 100% credential retrieval at K=16 where six baselines (H2O, TOVA, SnapKV, StreamingLLM, PyramidKV, DynamicKV) achieve 0%, and sustains 100% accuracy across 200 function-calling trials. TA-Fast, an attention-free variant, reduces memory overhead by 52% and is compatible with SDPA and FlashAttention. TA is orthogonal to existing compression methods and adds less than 1% latency overhead.

📄 PDF Abstract BibTeX arXiv:2604.11288

Code (0)

등록된 구현이 없습니다.

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

2026-07-12 · Venkatesha Matam, Keon Kim arxiv

Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a…

GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs

2026-05-29 · Junjie Peng, You Wu, Haoyi Wu, Jialong Han 외 arxiv

Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substantial memory overhead, motivating KV-cach…

Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

2026-05-10 · Ngoc Bui, Hieu Trung Nguyen, Arman Cohan, Rex Ying arxiv

The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative t…

Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs

2025-12-03 · Ngoc Bui, Shubham Sharma, Simran Lamba, Saumitra Mishra 외 arxiv

Memory and computation remain core bottlenecks in long-horizon LLM inference due to the quadratic cost of self-attention and the ever-growing key-value (KV) cache. Existing strategies for memory-bounded inference, such a…

Long-Context Understanding Mathematical Reasoning

Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation

2026-05-13 · Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li 외 arxiv

Autoregressive video generation enables streaming and open-ended long video synthesis, but still suffers from long-term degradation caused by accumulated errors. Existing KVCache strategies usually apply unified historic…

Video Generation