FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion. To mitigate this, practitioners typically rely on stale preconditioner updates, creating a fundamental trade-off between computational efficiency and optimization fidelity. In this work, we provide a theoretical study of staleness through the complementary lenses of convergence and stability. While staleness improves computational efficiency, it inherently degrades performance and introduces numerical instability. Crucially, we identify that damping, acting as a numerical stabilizer, can effectively suppress these negative effects. Guided by this analysis, we propose FOAM, an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency based on an approximation of the staleness-oriented error. Experimental results demonstrate that FOAM reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencySimilar Papers 제목 키워드 기반
Merging Memory and Space: A State Space Neural Operator
We propose the *State Space Neural Operator* (SS-NO), a compact architecture for learning solution operators of time-dependent partial differential equations (PDEs). Our formulation extends structured state space models …
Improving frequency response with synthetic damping available from fleets of distributed energy resources
With the increasing use of renewable generation in power systems, responsive resources will be necessary to support primary frequency control in future low-inertia/under-damped power systems. Flexible loads can provide f…
Spectral Audit of In-Context Operator Networks
Existing evaluations of neural operators and in-context operator learning rely primarily on prediction error, but accurate output prediction does not guarantee the correct local dynamical structure. A model may match sol…
Network-Reconfiguration-Aware Power Oscillation Damping Controller for Newly Commissioned Converter-Interfaced Power Plants
In recent years, transmission system operators have started requesting converter-interfaced generators (CIGs) to participate in grid services such as power oscillation damping (POD). As power systems are prone to topolog…
Foam-Agent: Towards Automated Intelligent CFD Workflows
Computational Fluid Dynamics (CFD) is an essential simulation tool in various engineering disciplines, but it often requires substantial domain expertise and manual configuration, creating barriers to entry. We present F…