Automatic Configuration for Optimal Communication Scheduling in DNN Training
ByteScheduler partitions and rearranges tensor transmissions to improve the communication efficiency of distributed Deep Neural Network (DNN) training. The configuration of hyper-parameters (i.e., the partition size and the credit size) is critical to the effectiveness of partitioning and rearrangement. Currently, ByteScheduler adopts Bayesian Optimization (BO) to find the optimal configuration for the hyper-parameters beforehand. In practice, however, various runtime factors (e.g., worker node status and network conditions) change over time, making the statically-determined one-shot configuration result suboptimal for real-world DNN training. To address this problem, we present a real-time configuration method (called AutoByte) that automatically and timely searches the optimal hyper-parameters as the training systems dynamically change. AutoByte extends the ByteScheduler framework with a meta-network, which takes the system's runtime statistics as its input and outputs predictions for speedups under specific configurations. Evaluation results on various DNN models show that AutoByte can dynamically tune the hyper-parameters with low resource usage, and deliver up to 33.2\% higher performance than the best static configuration in ByteScheduler.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationSchedulingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Downlink Clustering-Based Scheduling of IRS-Assisted Communications With Reconfiguration Constraints
Intelligent reflecting surfaces (IRSs) are being widely investigated as a potential low-cost and energy-efficient alternative to active relays for improving coverage in next-generation cellular networks. However, technic…
ClusteringQuantizationSchedulingDownlink TDMA Scheduling for IRS-aided Communications with Block-Static Constraints
Intelligent reflecting surfaces (IRSs) are being studied as possible low-cost energy-efficient alternatives to active relays, with the goal of solving the coverage issues of millimeter wave (mmWave) and terahertz (THz) n…
SchedulingPrediction-Assisted Online Distributed Deep Learning Workload Scheduling in GPU Clusters
The recent explosive growth of deep learning (DL) models has necessitated a compelling need for efficient job scheduling for distributed deep learning training with mixed parallelisms (DDLwMP) in GPU clusters. This paper…
GPUSchedulingAnalog-digital Scheduling for Federated Learning: A Communication-Efficient Approach
Over-the-air (OTA) computation has recently emerged as a communication-efficient Federated Learning (FL) paradigm to train machine learning models over wireless networks. However, its performance is limited by the device…
Federated LearningQuantizationSchedulingAutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs
Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation, with improvements scaling proportionally with model size. However, the limitations of GP…
CPUGPUSchedulingText Generation