Gradient Sparsification

2000년 도입 · 논문 38편에서 사용

Gradient Sparsification is a technique for distributed training that sparsifies stochastic gradients to reduce the communication cost, with minor increase in the number of iterations. The key idea behind our sparsification technique is to drop some coordinates of the stochastic gradient and appropriately amplify the remaining coordinates to ensure the unbiasedness of the sparsified stochastic gradient. The sparsification approach can significantly reduce the coding length of the stochastic gradient and only slightly increase the variance of the stochastic gradient.

출처: Gradient Sparsification for Communication-Efficient Distributed Optimization

소개 논문: Gradient Sparsification for Communication-Efficient Distributed Optimization