FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
Mixture-of-Experts (MoE) models enable scalable neural networks through conditional computation, offering enhanced effectiveness and efficiency for next-generation wireless communications. However, deploying MoE with federated learning (FL) over wireless and IoT edge networks faces two critical challenges: 1) resource-constrained clients cannot store large AI models with full expert sets, and 2) non-IID data distributions cause severe expert load imbalance that degrades model performance. To this end, we propose FLEX-MoE, a federated MoE framework that jointly optimizes expert assignment and load balancing under limited client capacity. Specifically, our approach introduces client-expert fitness scores that quantify expert suitability for local datasets through training feedback, and employs an optimization-based algorithm to maximize client-expert specialization while enforcing balanced expert utilization system-wide. Unlike greedy methods that focus solely on personalization while ignoring load imbalance, FLEX-MoE addresses expert utilization skew, which is particularly severe in heterogeneous edge FL. Our experimental results demonstrate superior accuracy and consistently balanced expert utilization across diverse resource-constrained scenarios for edge computing.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningSimilar Papers 제목 키워드 기반
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are …
Federated LearningMixture-of-ExpertsPrompt LearningRepresentation LearningFlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
Recent advances in mixture-of-experts architectures have shown that individual experts models can be trained federatedly, i.e., in isolation from other experts by using a common base model to facilitate coordination. How…
Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
Mixture-of-Experts (MoE) models are typically pre-trained with explicit load-balancing constraints to ensure statistically balanced expert routing. Despite this, we observe that even well-trained MoE models exhibit signi…
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
Heterogeneous LoRA-rank methods address system heterogeneity in federated fine-tuning of foundation models by assigning client-specific ranks based on computational capabilities. However, these methods achieve only margi…
Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
The integration of Federated Learning (FL) and Mixture-of-Experts (MoE) presents a compelling pathway for training more powerful, large-scale artificial intelligence models (LAMs) on decentralized data while preserving p…
Edge-computingFederated LearningMixture-of-Experts