홈 › Papers

When Does Muon Help Agentic Reinforcement Learning?

Muon is competitive with AdamW in large-scale pre-training, but its operating regime in reinforcement-learning post-training remains unclear. We map this regime on ALFWorld, a sparse-reward agentic benchmark, using three group-based objectives and Qwen2.5 models from 0.5B to 3B. Under a shared KL and clipping recipe, matched optimizer comparisons and AdamW learning-rate controls trace the usable step-size range. AdamW responds non-monotonically to its learning rate, whereas fan-in Muon remains stable at a more aggressive effective step. At a learning rate of 3 x 10^-5, it improves late success over an AdamW 10^-6 baseline after correction across rate-metric tests. Its normalized-AUC effect is directionally positive but less uniform; the heuristic-matched lower-rate effect is less consistent, and tuned AdamW nearly matches high-rate Muon on 3B GraphGPO. High-rate Muon applies 3.53 times AdamW's hidden-matrix update RMS, while a full-budget RMS-matched control removes the late-success gain. Together, these results identify a recipe-level operating regime in which fan-in Muon supports a more aggressive stable effective step under shared KL and clipping. The margin is largest when optimization headroom remains and contracts near saturation, after AdamW tuning, or under magnitude matching. The scale-matched control ties this spectral effect to Muon's scaling convention rather than establishing a universal optimizer ranking.

Code (0)

등록된 구현이 없습니다.

Tasks

Similar Papers 제목 키워드 기반