$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
Communication is a key bottleneck in distributed training. Recently, an \emph{error-compensated} compression technology was particularly designed for the \emph{centralized} learning and receives huge successes, by showing significant advantages over state-of-the-art compression based methods in saving the communication cost. Since the \emph{decentralized} training has been witnessed to be superior to the traditional \emph{centralized} training in the communication restricted scenario, therefore a natural question to ask is "how to apply the error-compensated technology to the decentralized learning to further reduce the communication cost." However, a trivial extension of compression based centralized training algorithms does not exist for the decentralized scenario. key difference between centralized and decentralized training makes this extension extremely non-trivial. In this paper, we propose an elegant algorithmic design to employ error-compensated stochastic gradient descent for the decentralized scenario, named $\texttt{DeepSqueeze}$. Both the theoretical analysis and the empirical study are provided to show the proposed $\texttt{DeepSqueeze}$ algorithm outperforms the existing compression based decentralized learning algorithms. To the best of our knowledge, this is the first time to apply the error-compensated compression to the decentralized learning.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Timing Recovery for Non-Orthogonal Multiple Access with Asynchronous Clocks
A passive optical network (PON) based on non-orthogonal multiple access (NOMA) meets low latency and high capacity. In the NOMA-PON, the asynchronous clocks between the strong and weak optical network units (ONUs) cause …
Early Human Capital Accumulation and Decentralization
Decentralization is a centerpiece in Cameroonian's government institutions' design. This chapter elaborates a simple hierarchy model for the analysis of the effects of power devolution. The model predicts overall positiv…
SoK: Blockchain Decentralization
Blockchain introduces decentralized trust in peer-to-peer networks, advancing security and democratizing systems. Yet, a unified definition for decentralization remains elusive. Our Systematization of Knowledge (SoK) see…
Causal InferenceDescriptiveDPO Meets PPO: Reinforced Token Optimization for RLHF
In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep rein…
Deep Reinforcement Learningreinforcement-learningSentenceDoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
A standard approach in large scale machine learning is distributed stochastic gradient training, which requires the computation of aggregated stochastic gradients over multiple nodes on a network. Communication is a majo…