paper-with-me

홈 › Papers

Upcycling Large Language Models into Mixture of Experts

2024-10-10 · Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, Bryan Catanzaro

Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models.

📄 PDF Abstract BibTeX arXiv:2410.07524

Code (0)

등록된 구현이 없습니다.

Tasks

Mixture-of-ExpertsMMLU

Methods 이 논문이 사용한 방법론

MoE 설명 없음

Similar Papers 제목 키워드 기반

Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling

2025-08-31 · Junfeng Ran, Guangxiang Zhao, Yuhan Wu, Dawei Zhu 외 arxiv

The Mixture-of-Experts (MoE) models have gained significant attention in deep learning due to their dynamic resource allocation and superior performance across diverse tasks. However, efficiently training these models re…

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging

2024-10-02 · Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Yu Sun 외

Mixture-of-Experts (MoE) shines brightly in large language models (LLMs) and demonstrates outstanding performance in plentiful natural language processing tasks. However, existing methods transforming LLMs from dense to …

DiversityMixture-of-Experts

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

2024-08-15 · Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo 외

The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibi…

Mixture-of-Experts

SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

2026-06-15 · Weiqiao Shan, Ruixiang Mao, Yuang Li, Yuhao Zhang 외 arxiv

Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. Ho…

Speech-to-Text Translation

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts

2024-08-28 · Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker 외

Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models. The Mixture of Experts (MoE) architecture has been the focus of significant …

Mixture-of-Experts