OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

2026-02-05 · Jingze Shi, Zhangyang Peng, Yizhang Zhu, Yifan Wu, Guang Liu, Yuyu Luo arxiv

Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution efficiency. We propose OmniMoE, a system-algorithm co-designed framework that pushes expert granularity to its logical extreme. OmniMoE introduces vector-level Atomic Experts, enabling scalable routing and execution within a single MoE layer, while retaining a shared dense MLP branch for general-purpose processing. Although this atomic design maximizes capacity, it poses severe challenges for routing complexity and memory access. To address these, OmniMoE adopts a system-algorithm co-design: (i) a Cartesian Product Router that decomposes the massive index space to reduce routing complexity from O(N) to O(sqrt(N)); and (ii) Expert-Centric Scheduling that inverts the execution order to turn scattered, memory-bound lookups into efficient dense matrix operations. Validated on seven benchmarks, OmniMoE (with 1.7B active parameters) achieves 50.9% zero-shot accuracy across seven benchmarks, outperforming coarse-grained (e.g., DeepSeekMoE) and fine-grained (e.g., PEER) baselines. Crucially, OmniMoE reduces inference latency from 73ms to 6.7ms (a 10.9-fold speedup) compared to PEER, demonstrating that massive-scale fine-grained MoE can be fast and accurate. Our code is open-sourced at https://github.com/flash-algo/omni-moe.

📄 PDF Abstract BibTeX arXiv:2602.05711

Code (0)

등록된 구현이 없습니다.

Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion

2025-11-18 · Ye Liu, Xu Chen, Wuji Chen, Mang Li arxiv

In cross-border e-commerce, search relevance modeling faces the dual challenge of extreme linguistic diversity and fine-grained semantic nuances. Existing approaches typically rely on scaling up a single monolithic Large…

Deep Mixture of Diverse Experts for Large-Scale Visual Recognition

2017-06-24 · Tianyi Zhao, Jun Yu, Zhenzhong Kuang, Wei zhang 외

In this paper, a deep mixture of diverse experts algorithm is developed for seamlessly combining a set of base deep CNNs (convolutional neural networks) with diverse outputs (task spaces), e.g., such base deep CNNs are t…

Multi-Task Learning Object Object Recognition

Mixture of experts architectures for machine learning interatomic potentials

2026-03-09 · Yuzhi Liu, Duo Zhang, Anyang Peng, Weinan E 외 arxiv

Machine Learning Interatomic Potentials (MLIPs) enable accurate large-scale atomistic simulations, yet improving their expressive capacity efficiently remains challenging. Here we systematically investigate Mixture-of-Ex…

AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

2025-05-19 · Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen 외

Machine learning-based interatomic potentials and force fields depend critically on accurate atomic structures, yet such data are scarce due to the limited availability of experimentally resolved crystals. Although atomi…

Denoising Formation Energy Property Prediction

HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space

2025-09-26 · Ke Li, Zheng Yang, Zhongbin Zhou, Feng Xue 외 arxiv

Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memor…