An SDE for Modeling SAM: Theory and Insights
We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent. Our main contribution is the derivation of continuous-time models (in the form of SDEs) for SAM and two of its variants, both for the full-batch and mini-batch settings. We demonstrate that these SDEs are rigorous approximations of the real discrete-time algorithms (in a weak sense, scaling linearly with the learning rate). Using these models, we then offer an explanation of why SAM prefers flat minima over sharp ones~--~by showing that it minimizes an implicitly regularized loss with a Hessian-dependent noise structure. Finally, we prove that SAM is attracted to saddle points under some realistic conditions. Our theoretical results are supported by detailed experiments.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Function Alignment: A New Theory of Mind and Intelligence, Part I: Foundations
This paper introduces function alignment, a novel theory of mind and intelligence that is both intuitively compelling and structurally grounded. It explicitly models how meaning, interpretation, and analogy emerge from i…
On the Computation and Applications of Large Dense Partial Correlation Networks
While sparse inverse covariance matrices are very popular for modeling network connectivity, the value of the dense solution is often overlooked. In fact the L2-regularized solution has deep connections to a number of im…
Dimensionality ReductionModel SelectionUncertainty QuantificationPosition: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling
Time series (TS) modeling has come a long way from early statistical, mainly linear, approaches to the current trend in TS foundation models. With a lot of hype and industrial demand in this field, it is not always clear…
Conceptive Artificial Intelligence: Insights from design theory
The current paper offers a perspective on what we term conceptive intelligence - the capacity of an agent to continuously think of new object definitions (tasks, problems, physical systems, etc.) and to look for methods …
Seeing Through Risk: A Symbolic Approximation of Prospect Theory
We propose a novel symbolic modeling framework for decision-making under risk that merges interpretability with the core insights of Prospect Theory. Our approach replaces opaque utility curves and probability weighting …
Decision Making