Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed algorithm significantly reduces the communication overhead, which, in turn, reduces the training time by up to 30% for the 1B word dataset.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency
Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes. Despite the recent theoretical advances of local SGD in empir…
Communication-Efficient Distributed SGD with Compressed Sensing
We consider large scale distributed optimization over a set of edge devices connected to a central server, where the limited communication bandwidth between the server and edge devices imposes a significant bottleneck fo…
compressed sensingDistributed OptimizationFederated LearningA Communication Efficient Collaborative Learning Framework for Distributed Features
We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular…
STL-SGD: Speeding Up Local SGD with Stagewise Communication Period
Distributed parallel stochastic gradient descent algorithms are workhorses for large scale machine learning tasks. Among them, local stochastic gradient descent (Local SGD) has attracted significant attention due to its …
Flattened one-bit stochastic gradient descent: compressed distributed optimization with controlled variance
We propose a novel algorithm for distributed stochastic gradient descent (SGD) with compressed gradient communication in the parameter-server framework. Our gradient compression technique, named flattened one-bit stochas…
Distributed OptimizationQuantization