Dynamic Scheduling of MPI-based Distributed Deep Learning Training Jobs
There is a general trend towards solving problems suited to deep learning with more complex deep learning architectures trained on larger training sets. This requires longer compute times and greater data parallelization or model parallelization. Both data and model parallelism have been historically faster in parameter server architectures, but data parallelism is starting to be faster in ring architectures due to algorithmic improvements. In this paper, we analyze the math behind ring architectures and make an informed adaptation of dynamic scheduling to ring architectures. To do so, we formulate a non-convex, non-linear, NP-hard integer programming problem and a new efficient doubling heuristic for its solution. We build upon Horovod: an open source ring architecture framework over TensorFlow. We show that Horovod jobs have a low cost to stop and restart and that stopping and restarting ring architecture jobs leads to faster completion times. These two facts make dynamic scheduling of ring architecture jobs feasible. Lastly, we simulate a scheduler using these runs and show a more than halving of average job time on some workload patterns.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningMathSchedulingSimilar Papers 제목 키워드 기반
A Deep Reinforcement Learning Approach to Multi-component Job Scheduling in Edge Computing
We are interested in the optimal scheduling of a collection of multi-component application jobs in an edge computing system that consists of geo-distributed edge computing nodes connected through a wide area network. The…
Deep Reinforcement LearningEdge-computingreinforcement-learningReinforcement Learning+2Prediction-Assisted Online Distributed Deep Learning Workload Scheduling in GPU Clusters
The recent explosive growth of deep learning (DL) models has necessitated a compelling need for efficient job scheduling for distributed deep learning training with mixed parallelisms (DDLwMP) in GPU clusters. This paper…
GPUSchedulingCommunication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
Distributed Deep Learning (DDL) has rapidly grown its popularity since it helps boost the training performance on high-performance GPU clusters. Efficient job scheduling is indispensable to maximize the overall performan…
Deep LearningGPUSchedulingVenn: Resource Management for Collaborative Learning Jobs
In recent years, collaborative learning (CL) has emerged as a promising approach for machine learning (ML) and data science across distributed edge devices. As the deployment of CL jobs increases, they inevitably contend…
Federated LearningManagementSchedulingReducing Fragmentation and Starvation in GPU Clusters through Dynamic Multi-Objective Scheduling
GPU clusters have become essential for training and deploying modern AI systems, yet real deployments continue to report average utilization near 50%. This inefficiency is largely caused by fragmentation, heterogeneous w…