홈 › Papers

Muon is Scalable for LLM Training

2025-02-24 · Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

📄 PDF Abstract BibTeX arXiv:2502.16982

Code (1)

KellerJordan/Muon 공식 구현 pytorch

Tasks

Computational Efficiency

Methods 이 논문이 사용한 방법론

Weight Decay 설명 없음

AdamW AdamW is a stochastic optimization method that modifies the typical implementation of weight decay in Adam, by decoupling [weight…

MuonRec: Shifting the Optimizer Paradigm Beyond Adam in Scalable Generative Recommendation

2026-02-28 · Rong Shan, Aofan Yu, Bo Chen, Kuo Cai 외 arxiv

Recommender systems (RecSys) are increasingly emphasizing scaling, leveraging larger architectures and more interaction data to improve personalization. Yet, despite the optimizer's pivotal role in training, modern RecSy…

NorMuon: Making Muon more efficient and scalable

2025-10-07 · Zichong Li, Liming Liu, Chen Liang, Weizhu Chen 외 arxiv

The choice of optimizer significantly impacts the training efficiency and computational costs of large language models (LLMs). Recently, the Muon optimizer has demonstrated promising results by orthogonalizing parameter …

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

2026-05-26 · Jiacheng Li, Jianchao Tan, Hongtao Xu, Jiaqi Zhang 외 arxiv

The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Mu…

Mathematical Reasoning Code Generation

MuonSSM: Orthogonalizing State Space Models for Sequence Modeling

2026-06-29 · Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo, Tan M. Nguyen 외 arxiv

State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due…

HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

2026-03-10 · Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng 외 arxiv

Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and o…

Image Classification