Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsel…
Mixture-of-ExpertsA Theory of Ex Post Rationalization
People rationalize their past choices, even those that were mistakes in hindsight. We propose a formal theory of this behavior. The theory predicts that sunk costs affect later choices. Its model primitives are identifie…
The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collected data is informative and aligned with…
Deep Reinforcement LearningMuJoCoReinforcement Learning (RL)Asymmetric All-Pay Auctions with Spillovers
When opposing parties compete for a prize, the sunk effort players exert during the conflict can affect the value of the winner's reward. These spillovers can have substantial influence on the equilibrium behavior of par…
AllMarketingDisUnknown: Distilling Unknown Factors for Disentanglement Learning
Disentangling data into interpretable and independent factors is critical for controllable generation tasks. With the availability of labeled data, supervision can help enforce the separation of specific factors as expec…
Disentanglement