AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

2024-06-19 · Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, Zhijie Deng

Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significant overheads. Nevertheless, existing MoE methods usually enforce a constant top-k routing for all tokens, which is arguably restrictive because various tokens (e.g., "<EOS>" vs. "apple") may require various numbers of experts for feature abstraction. Lifting such a constraint can help make the most of limited resources and unleash the potential of the model for downstream tasks. In this sense, we introduce AdaMoE to realize token-adaptive routing for MoE, where different tokens are permitted to select a various number of experts. AdaMoE makes minimal modifications to the vanilla MoE with top-k routing -- it simply introduces a fixed number of null experts, which do not consume any FLOPs, to the expert set and increases the value of k. AdaMoE does not force each token to occupy a fixed number of null experts but ensures the average usage of the null experts with a load-balancing loss, leading to an adaptive number of null/true experts used by each token. AdaMoE exhibits a strong resemblance to MoEs with expert choice routing while allowing for trivial auto-regressive modeling. AdaMoE is easy to implement and can be effectively applied to pre-trained (MoE-)LLMs. Extensive studies show that AdaMoE can reduce average expert load (FLOPs) while achieving superior performance. For example, on the ARC-C dataset, applying our method to fine-tuning Mixtral-8x7B can reduce FLOPs by 14.5% while increasing accuracy by 1.69%.

📄 PDF Abstract BibTeX arXiv:2406.13233

Code (1)

cengzihao/adamoe 공식 구현 pytorch

Tasks

ARCMixture-of-Experts

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

MoE 설명 없음

Improving MoE Compute Efficiency by Composing Weight and Data Sparsity

2026-01-21 · Maciej Kilian, Oleg Mkrtchyan, Luke Zettlemoyer, Akshat Shrivastava 외 arxiv

Mixture-of-Experts layers achieve compute efficiency through weight sparsity: each token activates only a subset of experts. Data sparsity, where each expert processes only a subset of tokens, offers a complementary axis…

TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation

2026-07-07 · Andrii Balashov, Olena Ponomarova arxiv

Conditional computation can decouple language model quality from per-token inference cost, yet leading techniques act on a single axis in isolation: Mixture-of-Experts (MoE) sparsifies the FFN, Mixture-of-Depths (MoD) sk…

LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts

2025-09-30 · Yuan Zhuang, Yi Shen, Yuexin Bian, Qing Su 외 arxiv

Recent studies have shown that combining parameter-efficient fine-tuning (PEFT) with mixture-of-experts (MoE) is an effective strategy for adapting large language models (LLMs) to the downstream tasks. However, most exis…

parameter-efficient fine-tuning

GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

2026-06-24 · Chaoxiang Cai, Minghe Weng, Jie Li, Yibo Jiang 외 arxiv

With the increase in model parameters and training data, the instruction following and generalization capabilities of Large VisionLanguage Models (LVLMs) have been significantly improved. Based on the Mixture of Experts …

Instruction Following

Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts

2026-05-06 · Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf, Michael Muehlebach arxiv

Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token. Recent evidence suggests that performance improves with increasingly granular experts, i.e., man…