Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving

2026-06-01 · Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami, Mingyuan Xiang, Henry Hoffmann arxiv

LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives. Existing multi-turn systems keep the turn as the scheduling unit and decide, turn by turn, whether to disaggregate prefill from decode. That decision rests on the turn's decode length, tool behavior, and KV growth, quantities that are not observable when the scheduler must act, forcing the system to predict them. We show this dependence on prediction is imposed by the scheduling unit, not the workload. Raising the scheduling unit from the turn to the conversation converts turn-level irregularity into a stable, two-phase structure: 1) a compute-bound turn-1 prefill followed by 2) a long, memory-bound tail. Thus, with the conversation as the scheduling unit, placement reduces to reading the first-turn input length and per-decoder KV occupancy, both directly observable. We instantiate this principle in ConServe, which routes the first-turn prefill to a high-throughput prefiller, transfers the KV cache exactly once, and pins the conversation to a single decoder for its entire tail, with no learned model of decode-side cost. Against a per-turn prediction baseline, ConServe reduces p95 time-to-first-effective-token (the latency of a conversation's first user-visible output) by 51.08% and improves energy efficiency by 7.51% while preserving last-turn TBT and SLOs; mapping the two phases onto heterogeneous GPU tiers adds a further 22.75% in energy efficiency.

📄 PDF Abstract BibTeX arXiv:2606.01839

Code (0)

등록된 구현이 없습니다.

semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage

2025-04-28 · Ke Hong, Lufang Chen, Zhong Wang, Xiuhong Li 외

Existing large language model (LLM) serving systems fall into two categories: 1) a unified system where prefill phase and decode phase are co-located on the same GPU, sharing the unified computational resource and storag…

GPU Large Language Model Scheduling

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

2025-12-25 · Xinglin Pan, Shaohuai Shi, Wenxiang Lin, Yuxin Wang 외 arxiv

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent disaggregated expert par…

Optimizing Attention on GPUs by Exploiting GPU Architectural NUMA Effects

2025-11-03 · Mansi Choudhary, Karthik Sangaiah, Sonali Singh, Muhammad Osama 외 arxiv

The rise of disaggregated AI GPUs has exposed a critical bottleneck in large-scale attention workloads: non-uniform memory access (NUMA). As multi-chiplet designs become the norm for scaling compute capabilities, memory …

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling

2025-04-03 · Weiqing Li, Guochao Jiang, Xiangyong Ding, Zhangcheng Tao 외

Disaggregated inference has become an essential framework that separates the prefill (P) and decode (D) stages in large language model inference to improve throughput. However, the KV cache transfer faces significant del…

Language Modeling Language Modelling Large Language Model Scheduling

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

2026-05-20 · Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang 외 arxiv

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL roll…