Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

2024-02-22 · Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li

A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

📄 PDF Abstract BibTeX arXiv:2402.14800

Code (1)

lucky-lance/expert_sparsity 공식 구현 pytorch

Tasks

AllMixture-of-Experts

Methods 이 논문이 사용한 방법론

Pruning 설명 없음

MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping

2025-11-19 · Yushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding 외 arxiv

Mixture-of-Experts (MoE) Multimodal large language models (MLLMs) excel at vision-language tasks, but they suffer from high computational inefficiency. To reduce inference overhead, expert skipping methods have been prop…

Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs

2026-01-31 · Hao Mark Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang 외 arxiv

Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) a…

Graph Sparsification via Mixture of Graphs

2024-05-23 · Guibin Zhang, Xiangguo Sun, Yanwei Yue, Chonghe Jiang 외

Graph Neural Networks (GNNs) have demonstrated superior performance across various graph learning tasks but face significant computational challenges when applied to large-scale graphs. One effective approach to mitigate…

Graph Learning Mixture-of-Experts

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

2026-07-02 · Yongqin Zeng, Sicheng Pan, Jiale Wang, Hai-tao Zheng 외 arxiv

Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-prunin…

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

2024-10-08 · Wei Huang, Yue Liao, Jianhui Liu, Ruifei He 외

Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consu…

Mixture-of-Experts Quantization