Deterministic Denominator Design for Localized Tamed Stochastic Gradient Langevin Dynamics
If the denominator in a tamed stochastic gradient Langevin update uses the current stochastic-gradient draw, the conditional mean can be biased even when the stochastic-gradient oracle is unbiased. A state-dependent denominator fixed before that draw removes this coupling. We build practical deterministic denominators from a short pilot run. A log-scale proxy is fitted to the growth score $G_\star(x)=\|b(x)\|/(1+\|x\|)$, and empirical pilot quantiles set the activation thresholds of a local proxy-quantile envelope. The reported proxy-quantile experiments use this local denominator. Separately, we describe a final denominator with a norm-polynomial tail-floor correction that can be used when one wants to certify the global effective-linearity input required by the companion deterministic-envelope Lyapunov theory. We show how proxy and threshold errors enter denominator errors and the resulting stationary observable errors. In the reported experiments, the local proxy-quantile denominator improves over random-denominator tamed SGLD at comparable production cost. It also gives observable behavior close to the $G_\star$-envelope benchmark, without the full-gradient growth-score evaluations required by that benchmark.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming
Stochastic gradient Langevin algorithms often use tamed denominators to stabilize superlinear drifts. This paper shows that when the denominator depends on the current stochastic gradient, the transformed update can have…
RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning
We introduce RELTA-SGLD, a taming scheme that stabilizes superlinear stochastic-gradient updates while reducing unnecessary suppression of the original learning drift. A threshold determines where the taming turns on, wh…
Tamed Stochastic Gradient Hamiltonian Monte Carlo
In this paper, we propose a novel tamed stochastic gradient Hamiltonian Monte Carlo (tSGHMC) algorithm for sampling and stochastic optimization problems with superlinearly growing stochastic gradients. Under a certain co…
Stochastic OptimizationTaming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function h…
Error estimates for tamed Euler and Randomized Euler schemes for SDEs with locally Lipschitz drift with applications to non-logconcave sampling and optimization
In this paper, we study the numerical discretization of stochastic differential equations with locally Lipschitz, super-linearly growing drift, and the resulting implications for sampling from non-log-concave distributio…