A Study of Gradient Variance in Deep Learning
The impact of gradient noise on training deep models is widely acknowledged but not well understood. In this context, we study the distribution of gradients during training. We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We prove that the variance of average mini-batch gradient is minimized if the elements are sampled from a weighted clustering in the gradient space. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training, and smaller learning rates coincide with higher variance. In addition, we introduce normalized gradient variance as a statistic that better correlates with the speed of convergence compared to gradient variance.
Code (1)
Tasks
ClusteringDeep LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Impact of the Mini-batch Size on the Dynamics of SGD: Variance and Beyond
We study mini-batch stochastic gradient descent (SGD) dynamics under linear regression and deep linear networks by focusing on the variance of the gradients only given the initial weights and mini-batch size, which is th…
regressionThe Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent
The mini-batch stochastic gradient descent (SGD) algorithm is widely used in training machine learning models, in particular deep learning models. We study SGD dynamics under linear regression and two-layer linear networ…
regressionAnalyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive …
Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference
Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that th…
Bayesian InferenceVariational InferenceAn Investigation of the Bias-Variance Tradeoff in Meta-Gradients
Meta-gradients provide a general approach for optimizing the meta-parameters of reinforcement learning (RL) algorithms. Estimation of meta-gradients is central to the performance of these meta-algorithms, and has been st…
Meta-LearningReinforcement Learning (RL)