Non-Gaussianity of Stochastic Gradient Noise
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A note on diffusion limits for stochastic gradient descent
In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in st…
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption …
Max-affine regression via first-order methods
We consider regression of a max-affine model that produces a piecewise linear model by combining affine models via the max function. The max-affine model ubiquitously arises in applications in signal processing and stati…
regressionRetrievalGaussDetect-LiNGAM:Causal Direction Identification without Gaussianity test
We propose GaussDetect-LiNGAM, a novel approach for bivariate causal discovery that eliminates the need for explicit Gaussianity tests by leveraging a fundamental equivalence between noise Gaussianity and residual indepe…
Causal InferenceThe amplitude modulation pattern of Gaussian noise is a fingerprint of Gaussianity
We introduce a new approach for Gaussianity testing using the envelope of a signal and its coefficient of variation. The envelope of a Gaussian signal follows the Rayleigh distribution, and given that the coefficient of …
Time SeriesTime Series Analysis