홈 › Papers

HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs

2025-04-04 · Yongji Wu, Xueshen Liu, Shuowei Jin, Ceyu Xu, Feng Qian, Z. Morley Mao, Matthew Lentz, Danyang Zhuo, Ion Stoica

The Mixture-of-Experts (MoE) architecture has become increasingly popular as a method to scale up large language models (LLMs). To save costs, heterogeneity-aware training solutions have been proposed to utilize GPU clusters made up of both newer and older-generation GPUs. However, existing solutions are agnostic to the performance characteristics of different MoE model components (i.e., attention and expert) and do not fully utilize each GPU's compute capability. In this paper, we introduce HeterMoE, a system to efficiently train MoE models on heterogeneous GPUs. Our key insight is that newer GPUs significantly outperform older generations on attention due to architectural advancements, while older GPUs are still relatively efficient for experts. HeterMoE disaggregates attention and expert computation, where older GPUs are only assigned with expert modules. Through the proposed zebra parallelism, HeterMoE overlaps the computation on different GPUs, in addition to employing an asymmetric expert assignment strategy for fine-grained load balancing to minimize GPU idle time. Our evaluation shows that HeterMoE achieves up to 2.3x speed-up compared to existing MoE training systems, and 1.4x compared to an optimally balanced heterogeneity-aware solution. HeterMoE efficiently utilizes older GPUs by maintaining 95% training throughput on average, even with half of the GPUs in a homogeneous A40 cluster replaced with V100.

📄 PDF Abstract BibTeX arXiv:2504.03871

Code (0)

등록된 구현이 없습니다.

Tasks

GPUMixture-of-Experts

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

MoE 설명 없음

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

2024-10-22 · Jialong Li, Shreyansh Tripathi, Lakshay Rastogi, Yiming Lei 외

As machine learning models scale in size and complexity, their computational requirements become a significant barrier. Mixture-of-Experts (MoE) models alleviate this issue by selectively activating relevant experts. Des…

All GPU Mixture-of-Experts Scheduling

MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models

2026-01-11 · Xin Ye, Daning Cheng, Boyang Zhang, Yunquan Zhang arxiv

Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardw…

Mixture of Heterogeneous Grouped Experts for Language Modeling

2026-04-25 · Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang 외 arxiv

Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enforce uniform expert sizes,creating a rig…

Scattered Mixture-of-Experts Implementation

2024-03-13 · Shawn Tan, Yikang Shen, Rameswar Panda, Aaron Courville

We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and overcoming some of the limitations to improve inference and training speed, and m…

Mixture-of-Experts

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

2025-04-21 · Dennis Liu, Zijie Yan, Xin Yao, Tong Liu 외

Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageable computation costs. However, efficient…