LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

2026-02-12 · Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li, Bin Cui arxiv

Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different input data. However, during expert parallel training, dynamic routing results in significant load imbalance among experts: a handful of overloaded experts hinder overall iteration, emerging as a training bottleneck. In this paper, we introduce LAER-MoE, an efficient MoE training framework. The core of LAER-MoE is a novel parallel paradigm, Fully Sharded Expert Parallel (FSEP), which fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. This allows for flexible re-layout of expert parameters during training to enhance load balancing. In particular, we perform fine-grained scheduling of communication operations to minimize communication overhead. Additionally, we develop a load balancing planner to formulate re-layout strategies of experts and routing schemes for tokens during training. We perform experiments on an A100 cluster, and the results indicate that our system achieves up to 1.69x acceleration compared to the current state-of-the-art training systems. Source code available at https://github.com/PKU-DAIR/Hetu-Galvatron/tree/laer-moe.

📄 PDF Abstract BibTeX arXiv:2602.11686

Code (0)

등록된 구현이 없습니다.

Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation

2025-12-18 · Zhenyu Liu, Yunzhen Liu, Zehao Fan, Garrett Gagnon 외 arxiv

Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers tha…

TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading

2026-03-01 · Yudong Pan, Yintao He, Tianhua Han, Lian Liu 외 arxiv

To deploy large Mixture-of-Experts (MoE) models cost-effectively, offloading-based single-GPU heterogeneous inference is crucial. While GPU-CPU architectures that offload cold experts are constrained by host memory bandw…

Tutel: Adaptive Mixture-of-Experts at Scale

2022-06-07 · Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang 외

Sparsely-gated mixture-of-experts (MoE) has been widely adopted to scale deep learning models to trillion-plus parameters with fixed computational cost. The algorithmic performance of MoE relies on its token routing mech…

Mixture-of-Experts Object Detection

A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router

2026-05-27 · O. M. Kiselev arxiv

We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts (MoE) layer. The model is obtained as a mean-field limit of a discrete reinforcement rule: the selected expert receives…

Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

2024-10-14 · Jun Luo, Chen Chen, Shandong Wu

Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are …

Federated Learning Mixture-of-Experts Prompt Learning Representation Learning