An asynchronous distributed training algorithm based on Gossip
Distributed training is widely used in deep learning, and distributed method is divided into centralized and decentralized. Centralized distributed clusters, such as Parameter Server (PS), have been widely used nowadays. The main problems of PS are the communication bottleneck and the security problem of the central node. For example, Horovod framework based on Ring AllReduce algorithm, one of the existing decentralized distributed clusters, can reduce the bottleneck of the central node communication. However, the framework is not completely decentralized, and the communication algorithm is synchronous, which will lead to longer communication waiting time inter-node in the cluster. Combined the Gossip protocol with Stochastic Gradient Descent (SGD), this paper proposes a communication framework Gossip Ring SGD (GR-SGD) for deep learning. GR-SGD is decentralized and asynchronous, and solves the problem of long communication waiting time. This paper uses the ImageNet data set and the ResNet model to verify the feasibility of the algorithm and compares it with Ring AllReduce and D-PSGD. Moreover, this paper also indicates that some data redundancy can reduce communication overhead and increase system fault tolerance.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How to scale distributed deep learning?
Training time on large datasets for deep neural networks is the principal workflow bottleneck in a number of important applications of deep learning, such as object classification and detection in automatic driver assist…
Deep LearningGeneral ClassificationAsynchronous Gossip Algorithms for Rank-Based Statistical Methods
As decentralized AI and edge intelligence become increasingly prevalent, ensuring robustness and trustworthiness in such distributed settings has become a critical issue-especially in the presence of corrupted or adversa…
GoSGD: Distributed Optimization for Deep Learning with Gossip Exchange
We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applyi…
Deep LearningDistributed OptimizationAsynchronous Bayesian Learning over a Network
We present a practical asynchronous data fusion model for networked agents to perform distributed Bayesian learning without sharing raw data. Our algorithm uses a gossip-based approach where pairs of randomly selected ag…
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
In this paper, we present GossipGraD - a gossip communication protocol based Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL) algorithms on large-scale systems. The salient features of GossipGra…
CPUDeep LearningGPU