PowerSGD

2000년 도입 · 논문 3편에서 사용

PowerSGD is a distributed optimization technique that computes a low-rank approximation of the gradient using a generalized power iteration (known as subspace iteration). The approximation is computationally light-weight, avoiding any prohibitively expensive Singular Value Decomposition. To improve the quality of the efficient approximation, the authors warm-start the power iteration by reusing the approximation from the previous optimization step.

출처: PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

소개 논문: PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization