GoSGD: Distributed Optimization for Deep Learning with Gossip Exchange
We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information between different threads based on gossip algorithms that show good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningDistributed OptimizationSimilar Papers 제목 키워드 기반
Gossip training for deep learning
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each appl…
Deep LearningAccelerated Gossip via Stochastic Heavy Ball Method
In this paper we show how the stochastic heavy ball method (SHB) -- a popular method for solving stochastic convex and non-convex optimization problems --operates as a randomized gossip algorithm. In particular, we focus…
Non asymptotic bounds in asynchronous sum-weight gossip protocols
This paper focuses on non-asymptotic diffusion time in asynchronous gossip protocols. Asynchronous gossip protocols are designed to perform distributed computation in a network of nodes by randomly exchanging messages on…
Consensus Is All You Need: Gossip-Based Reasoning Among Large Language Models
Large language models have advanced rapidly, but no single model excels in every area -- each has its strengths and weaknesses. Instead of relying on one model alone, we take inspiration from gossip protocols in distribu…
On the Limit Performance of Floating Gossip
In this paper we investigate the limit performance of Floating Gossip, a new, fully distributed Gossip Learning scheme which relies on Floating Content to implement location-based probabilistic evolution of machine learn…