Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

2026-05-07 · Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini, Philip Levis arxiv

Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TBT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.

📄 PDF Abstract BibTeX arXiv:2605.06206

Code (0)

등록된 구현이 없습니다.

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

2026-06-17 · Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob 외 arxiv

Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. Mixture-of-Experts (MoEs) architectures partially decouple model capacity from per-token …

Cooperation and Federation in Distributed Radar Point Cloud Processing

2024-05-03 · S. Savazzi, V. Rampa, S. Kianoush, A. Minora 외

The paper considers the problem of human-scale RF sensing utilizing a network of resource-constrained MIMO radars with low range-azimuth resolution. The radars operate in the mmWave band and obtain time-varying 3D point …

Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clients

2024-07-06 · Mengmeng Ma, Tang Li, Xi Peng

Federated Learning is widely employed to tackle distributed sensitive data. Existing methods primarily focus on addressing in-federation data heterogeneity. However, we observed that they suffer from significant performa…

Federated Learning Privacy Preserving

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

2024-01-16 · Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni 외

In large language models like the Generative Pre-trained Transformer, the Mixture of Experts paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, deploying GPT MoE models…

GPU Mixture-of-Experts

WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

2024-05-06 · Nan Xue, Yaping Sun, Zhiyong Chen, Meixia Tao 외

Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a …

Mixture-of-Experts