Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
The method of choice for parameter aggregation in Deep Neural Network (DNN) training, a network-intensive task, is shifting from the Parameter Server model to decentralized aggregation schemes (AllReduce) inspired by theoretical guarantees of better performance. However, current implementations of AllReduce overlook the interdependence of communication and computation, resulting in significant performance degradation. In this paper, we develop Caramel, a system that accelerates decentralized distributed deep learning through model-aware computation scheduling and communication optimizations for AllReduce. Caramel achieves this goal through (a) computation DAG scheduling that expands the feasible window of transfer for each parameter (transfer boundaries), and (b) network optimizations for smoothening of the load including adaptive batching and pipelining of parameter transfers. Caramel maintains the correctness of the dataflow model, is hardware-independent, and does not require any user-level or framework-level changes. We implement Caramel over TensorFlow and show that the iteration time of DNN training can be improved by up to 3.62x in a cloud environment.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSchedulingSimilar Papers 제목 키워드 기반
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computations. This has been commonly adopted in …
AllGPUSchedulingDynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling
To reduce uploading bandwidth and address privacy concerns, deep learning at the network edge has been an emerging topic. Typically, edge devices collaboratively train a shared model using real-time generated data throug…
SchedulingA Multi-Agent System Approach to Load-Balancing and Resource Allocation for Distributed Computing
In this research we use a decentralized computing approach to allocate and schedule tasks on a massively distributed grid. Using emergent properties of multi-agent systems, the algorithm dynamically creates and dissociat…
CPUDistributed ComputingSchedulingDecentralized Task Scheduling in Distributed Systems: A Deep Reinforcement Learning Approach
Efficient task scheduling in large-scale distributed systems presents significant challenges due to dynamic workloads, heterogeneous resources, and competing quality-of-service requirements. Traditional centralized appro…
Reinforcement LearningPopSGD: Decentralized Stochastic Gradient Descent in the Population Model
The population model is a standard way to represent large-scale decentralized distributed systems, in which agents with limited computational power interact in randomly chosen pairs, in order to collectively solve global…
Distributed OptimizationScheduling