홈 › Papers

ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models

2026-06-09 · Wenhao Liu, Hao Shi, Yunhe Li, Weizhi Fei, Xiangyuan Wang, Mengzhe Ruan, Hanxu Hou, Peisong Wang, Linqi Song, Shuang Qiu arxiv

Long chain-of-thought (CoT) trajectories in large language model (LLM) reasoning cause severe inference bottlenecks due to rapid key-value (KV) cache growth. Current decoding-time compression methods mitigate this issue via token eviction, but typically assume a uniform budget distribution across all layers and heads. In contrast, existing non-uniform budget allocation methods are predominantly designed for the static prompt prefill phase, and they do not capture the stepwise context demands of autoregressive reasoning. To bridge this gap, we propose ReasonAlloc, a training-free framework that recasts decoding-time KV compression as a hierarchical budget allocation problem. ReasonAlloc operates at two complementary levels: an offline layer-wise preallocation strategy captures an architecture-driven demand pattern which we call ``\textit{Reasoning Wave}'', while an online head-wise strategy reallocates resources during decoding to information-rich heads based on real-time utility. Evaluations on mathematical reasoning benchmarks (MATH-500, AIME~2024) using DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-14B, and AceReason-14B show that ReasonAlloc outperforms uniform-budget R-KV, SnapKV, and Pyramid-RKV (a baseline enforcing a static, monotonically decreasing layer budget), with the largest gains at small budgets (128-512 tokens). ReasonAlloc is plug-and-play with existing token-eviction policies and introduces negligible inference-time overhead.

📄 PDF Abstract BibTeX arXiv:2606.11164

Code (0)

등록된 구현이 없습니다.

Tasks

Mathematical Reasoning

HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference

2026-04-07 · Bowen Zeng, Feiyang Ren, Jun Zhang, Xiaoling Gu 외 arxiv

Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is hindered by the rapid growth of key-value (KV) caches. Each visual input expands into thousan…

BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding

2026-05-29 · Liang He, Jingbo Wen, Qishi Zhan, Yixiong Chen 외 arxiv

Speculative decoding speeds up autoregressive decoding by using a drafter to propose multiple tokens that a verifier validates in parallel. In resource-constrained deployments, the drafter uses a sparse KV cache to limit…

QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

2025-02-05 · Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Hooper 외

Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache …

GPU

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

2025-05-24 · YiXuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu 외

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challen…

EntmaxKV: Support-Aware Decoding for Entmax Attention

2026-05-20 · Gonçalo Duarte, Miguel Couceiro, Marcos V. Treviso arxiv

Long-context decoding is increasingly limited by KV-cache memory traffic since each generated token attends over a cache whose size grows linearly with context length. Existing sparse decoding methods reduce this cost by…