SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning

2025-08-08 · Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang arxiv

Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to $\mathbf{2.53\times}$ time-to-first-token (TTFT) speedup and $\mathbf{1.88\times}$ end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code is available at https://github.com/Longxmas/SlimInfer.

📄 PDF Abstract BibTeX arXiv:2508.06447

Code (0)

등록된 구현이 없습니다.

MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention

2025-04-22 · Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu 외

The integration of long-context capabilities with visual understanding unlocks unprecedented potential for Vision Language Models (VLMs). However, the quadratic attention complexity during the pre-filling phase remains a…

GPU

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

2024-07-02 · Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu 외

The computational challenges of Large Language Model (LLM) inference remain a significant barrier to their widespread deployment, especially as prompt lengths continue to increase. Due to the quadratic complexity of the …

GPU Language Modelling Large Language Model

Long-Context Inference with Retrieval-Augmented Speculative Decoding

2025-02-27 · Guanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li 외

The emergence of long-context large language models (LLMs) offers a promising alternative to traditional retrieval-augmented generation (RAG) for processing extensive documents. However, the computational overhead of lon…

Computational Efficiency RAG Retrieval Retrieval-augmented Generation+1

AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

2025-01-04 · Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao 외

Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping m…

DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

2026-04-21 · Jinyu Guo, Zhihan Zhang, Jiehui Xie, Md. Tamim Iqbal 외 arxiv

The quadratic computational complexity of the standard attention mechanism constitutes a fundamental bottleneck for large language models in long-context inference. While existing KV cache compression methods alleviate m…