Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PIONM: A Generalized Approach to Solving Density-Constrained Mean-Field Games Equilibrium under Modified Boundary Conditions
Neural network-based methods are effective for solving equilibria in Mean-Field Games (MFGs), particularly in high-dimensional settings. However, solving the coupled partial differential equations (PDEs) in MFGs limits t…
Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
The convergence of stochastic interacting particle systems in the mean-field limit to solutions of conservative stochastic partial differential equations is established, with optimal rate of convergence. As a second main…
Implicit regularisation in stochastic gradient descent: from single-objective to two-player games
Recent years have seen many insights on deep learning optimisation being brought forward by finding implicit regularisation effects of commonly used gradient-based optimisers. Understanding implicit regularisation can no…
Structured Stochastic Variational Inference
Stochastic variational inference makes it possible to approximate posterior distributions induced by large datasets quickly using stochastic optimization. The algorithm relies on the use of fully factorized variational d…
SensitivityStochastic OptimizationVariational InferenceNonlinear reserving and multiple contract modifications in life insurance
Life insurance cash flows become reserve dependent when contract conditions are modified during the contract term on condition that actuarial equivalence is maintained. As a result, insurance cash flows and prospective r…