Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

2025-12-25 · Xinglin Pan, Shaohuai Shi, Wenxiang Lin, Yuxin Wang, Zhenheng Tang, Wei Wang, Xiaowen Chu arxiv

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent disaggregated expert parallelism (DEP) distributes attention and experts to dedicated GPU groups but lacks support for shared experts and efficient task scheduling, limiting performance. We propose FinDEP, a fine-grained task scheduling algorithm for DEP that maximizes task overlap to improve MoE inference throughput. FinDEP introduces three innovations: 1) partitioning computation/communication into smaller tasks for fine-grained pipelining, 2) formulating a scheduling optimization supporting variable granularity and ordering, and 3) developing an efficient solver for this large search space. Experiments on four GPU systems with DeepSeek-V2 and Qwen3-MoE show FinDEP improves throughput by up to 1.61x over prior methods, achieving up to 1.24x speedup on a 32-GPU system.

📄 PDF Abstract BibTeX arXiv:2512.21487

Code (0)

등록된 구현이 없습니다.

Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling

2026-03-29 · Songchen Ma, Hongyi Li, Weihao Zhang, Yonghao Tan 외 arxiv

Mixture-of-Experts is a promising approach for edge AI with low-batch inference. Yet, on-device deployments often face limited on-chip memory and severe workload imbalance; the prevalent use of offloading further incurs …

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling

2025-04-03 · Weiqing Li, Guochao Jiang, Xiangyong Ding, Zhangcheng Tao 외

Disaggregated inference has become an essential framework that separates the prefill (P) and decode (D) stages in large language model inference to improve throughput. However, the KV cache transfer faces significant del…

Language Modeling Language Modelling Large Language Model Scheduling

KVDirect: Distributed Disaggregated LLM Inference

2024-12-13 · Shiyang Chen, Rain Jiang, Dezhi Yu, Jinlai Xu 외

Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to impr…

GPU Scheduling

Optimizing Attention on GPUs by Exploiting GPU Architectural NUMA Effects

2025-11-03 · Mansi Choudhary, Karthik Sangaiah, Sonali Singh, Muhammad Osama 외 arxiv

The rise of disaggregated AI GPUs has exposed a critical bottleneck in large-scale attention workloads: non-uniform memory access (NUMA). As multi-chiplet designs become the norm for scaling compute capabilities, memory …

Frontier: Simulating the Next Generation of LLM Inference Systems

2025-08-05 · Yicheng Feng, Xin Tan, Kin Hang Sew, Yimin Jiang 외 arxiv

Large Language Model (LLM) inference is growing increasingly complex with the rise of Mixture-of-Experts (MoE) models and disaggregated architectures that decouple components like prefill/decode (PD) or attention/FFN (AF…