SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference

2026-03-05 · Luchang Li, Dongfang Li, Bozhao Gong, Yu Zhang arxiv

Prefill-Decode (P/D) disaggregation has emerged as a widely adopted optimization strategy for Large Language Model (LLM) inference. However, there currently exists no well-established methodology for determining the optimal number of P/D hardware resources, subject to constraints on total throughput, service level objectives (SLOs), and request characteristics - specifically input and output lengths. To address this gap, we propose a hybrid approach that combines theoretical modeling with empirical benchmarking. First, we present a theoretical model for calculating P/D resource counts, which is based on total throughput requirements, request input and output lengths, as well as prefill and decode throughput. Then, to obtain the actual prefill and decode throughput under SLO constraints, we model the prefill process using M/M/1 queuing theory, deriving the achieved prefill throughput from the benchmarked maximum prefill throughput and Time-To-First-Token (TTFT). For the decode phase, we determine the decode batch sizes that meet Time-Per-Output-Token (TPOT) requirements and obtain the corresponding decode throughput through empirical measurements. Our experimental results demonstrate that the proposed method can accurately predict optimal P/D resource allocation in real-world LLM inference scenarios.

📄 PDF Abstract BibTeX arXiv:2603.04716

Code (0)

등록된 구현이 없습니다.

Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving

2025-07-09 · Xiaoxiang Shi, Colin Cai, Junjia Du, Zhihao Jia arxiv

Monolithic serving with chunked prefill improves GPU utilization by batching prefill and decode together, but suffers from fine-grained phase interference. Engine-level prefill-decode (PD) disaggregation avoids interfere…

RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation

2026-01-16 · Amna Masood, Pratishtha Gaur, Nuwan Jayasena arxiv

Two widely adopted techniques for LLM inference serving systems today are hybrid batching and disaggregated serving. A hybrid batch combines prefill and decode tokens of different requests in the same batch to improve re…

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

2024-10-23 · Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter 외

Each request in LLM inference goes through two phases: compute-bound prefill and memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid batching that combines the prefill and decode phases o…

GPU

Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

2026-07-31 · Liming Liu, Mingze Wang, Tuo Zhao arxiv

As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training. In typical serving, prompt prefill runs in parallel and is compute-bound, whereas autoreg…

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling

2025-04-03 · Weiqing Li, Guochao Jiang, Xiangyong Ding, Zhangcheng Tao 외

Disaggregated inference has become an essential framework that separates the prefill (P) and decode (D) stages in large language model inference to improve throughput. However, the KV cache transfer faces significant del…

Language Modeling Language Modelling Large Language Model Scheduling