vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

2024-05-07 · Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar

PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.

📄 PDF Abstract BibTeX arXiv:2405.04437

Code (1)

microsoft/vattention 공식 구현 pytorch

Tasks

GPUManagement

Methods 이 논문이 사용한 방법론

Fragmentation Given a pattern $P,$ that is more complicated than the patterns, we fragment $P$ into simpler patterns such that their exact count is known. In the subgraph GNN proposed earlier,…

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

2026-07-17 · Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa Misailovic arxiv

Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory re…

vAttention: Verified Sparse Attention

2025-10-07 · Aditya Desai, Kumar Krishna Agrawal, Shuo Yang, Alejandro Cuadron 외 arxiv

State-of-the-art sparse attention methods for reducing decoding latency fall into two main categories: approximate top-$k$ (and its extension, top-$p$) and recently introduced sampling-based estimation. However, these ap…

Efficient Memory Management for Large Language Model Serving with PagedAttention

2023-09-12 · Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng 외

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge an…

Language Modeling Language Modelling Large Language Model Management

DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs

2026-01-12 · Nayoung Choi, Jonathan Zhang, Jinho D. Choi arxiv

Large Language Models (LLMs) increasingly operate over long-form dialogues with frequent topic shifts. While recent LLMs support extended context windows, efficient management of dialogue history in practice is needed du…

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

2024-06-28 · Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim

Transformer-based large language models (LLMs) demonstrate impressive performance across various natural language processing tasks. Serving LLM inference for generating long contents, however, poses a challenge due to th…

Management Text Generation