Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference
Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging due to mixed workloads comprising latency-sensitive (LS) and best-effort (BE) jobs. Existing inference systems employ iteration-level first-come-first-served scheduling, causing head-of-line blocking when BE jobs delay LS jobs. We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models, featuring a fine-grained, priority-aware preemptive scheduler. QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT). Our approach removes iteration-level scheduling constraints, enabling the scheduler to preempt jobs at any layer based on priority. Evaluations on an Nvidia A100 GPU show that QLLM significantly improves performance. It reduces LS TTFT by an average of $65.5\times$ and meets the SLO at up to $7$ requests/sec, whereas the baseline fails to do so under the tested workload. Additionally, it cuts LS turnaround time by up to $12.8\times$ without impacting throughput. QLLM is modular, extensible, and seamlessly integrates with Hugging Face MoE models.
Code (0)
등록된 구현이 없습니다.
Tasks
BlockingGPUMixture-of-ExpertsSchedulingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhancing Adaptive Mixed-Criticality Scheduling with Deep Reinforcement Learning
Adaptive Mixed-Criticality (AMC) is a fixed-priority preemptive scheduling algorithm for mixed-criticality hard real-time systems. It dominates many other scheduling algorithms for mixed-criticality systems, but does so …
Deep Reinforcement Learningreinforcement-learningReinforcement LearningSchedulingTopology-aware Preemptive Scheduling for Co-located LLM Workloads
Hosting diverse large language model workloads in a unified resource pool through co-location is cost-effective. For example, long-running chat services generally follow diurnal traffic patterns, which inspire co-locatio…
Language ModelingLanguage ModellingLarge Language ModelSchedulingFIKIT: Priority-Based Real-time GPU Multi-tasking Scheduling with Kernel Identification
Highly parallelized workloads like machine learning training, inferences and general HPC tasks are greatly accelerated using GPU devices. In a cloud computing cluster, serving a GPU's computation power through multi-task…
Cloud ComputingGPUSchedulingFixed Priority Global Scheduling from a Deep Learning Perspective
Deep Learning has been recently recognized as one of the feasible solutions to effectively address combinatorial optimization problems, which are often considered important yet challenging in various research domains. In…
Combinatorial OptimizationDeep LearningSchedulingOptimization of Traffic Control in MMAP[c]/PH[c]/S Catastrophic Queueing Model with PH Retrial Times and Controllable Preemptive Repeat Priority Policy
The presented study elaborates a multi-server catastrophic retrial queueing model considering preemptive repeat priority policy with phase-type (PH) distributed retrial times. For the sake of comprehension, the scenario …