MuonAll: Muon Variant for Efficient Finetuning of Large Language Models

2025-11-08 · Saurabh Page, Advait Joshi, S. S. Sonawane arxiv

Muon optimizer has demonstrated robust results in pretraining of language models but its performance in finetuning of existing public pretrained models is not yet explored. Currently, Muon is used along with AdamW introducing a scope of improvement for adopting all parameters inside Muon. We introduce MuonAll, which incorporates all the parameters inside Muon by transforming into 2D matrices. We conduct extensive finetuning experiments across publicly available language models with model sizes upto half billion parameters. Muon and MuonAll perform at par with AdamW across major benchmarks, highlighting their effectiveness as alternative optimizers. We open-source the distributed implementations of Muon and MuonAll, available at https://github.com/Saurabh750/optimizer

📄 PDF Abstract BibTeX arXiv:2511.06086

Code (0)

등록된 구현이 없습니다.

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

2026-05-07 · Yuxing Liu, Jianyu Wang, Tong Zhang arxiv

Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretrain…

Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds

2026-05-10 · Yibang Li, Bihari Lal Pandey, Ravi Sah, Andi Han 외 arxiv

Muon and related norm-constrained matrix optimizers have become central to large-scale learning problems. They are formulated as a linear maximization oracle (LMO) over an ambient matrix-norm ball in unconstrained Euclid…

Image Classification

LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

2026-06-11 · Franz Louis Cesista, Katherine Crowson, Cédric Simal, Stella Biderman arxiv

Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wise optimizers such as AdamW, it is sensit…

LiMuon: Light and Fast Muon Optimizer for Large Models

2025-09-18 · Feihu Huang, Yuning Luo, Songcan Chen arxiv

Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention. More recently, the useful Muon optimizer is specifically designed for matrix-structur…

Stochastic Optimization

Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum

2026-01-21 · Jingru Li, Yibo Fan, Huan Li arxiv

Large Language Models (LLMs) achieve competitive performance across diverse natural language processing (NLP) tasks, yet pretraining is computationally demanding, making optimizer efficiency an important practical consid…