ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's heterogeneous and diverse network fabrics. We present ForestColl, a tool that generates throughput-optimal schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretical optimality. Its schedule generation runs in strongly polynomial time and is highly scalable. ForestColl supports any network fabrics, including both switching fabrics and direct accelerator connections. We evaluated ForestColl on multi-box AMD MI250 and NVIDIA DGX A100 platforms. ForestColl showed significant improvements over the vendors' own optimized communication libraries, RCCL and NCCL, across various settings and in LLM training. ForestColl also outperformed other state-of-the-art schedule generation techniques with both more efficient generated schedules and substantially faster schedule generation speed.
Code (1)
Similar Papers 제목 키워드 기반
HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters
Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics…
Resource Allocation in Heterogeneously-Distributed Joint Radar-Communications under Asynchronous Bayesian Tracking Framework
Optimal allocation of shared resources is key to deliver the promise of jointly operating radar and communications systems. In this paper, unlike prior works which examine synergistic access to resources in colocated joi…
Heterogeneously-Distributed Joint Radar Communications: Bayesian Resource Allocation
Due to spectrum scarcity, the coexistence of radar and wireless communication has gained substantial research interest recently. Among many scenarios, the heterogeneouslydistributed joint radar-communication system is pr…
Joint Radar-CommunicationScheduling Optimization of Heterogeneous Services by Resolving Conflicts
Fifth generation (5G) new radio introduced flexible numerology to provide the necessary flexibility for accommodating heterogeneous services. However, optimizing the scheduling of heterogeneous services with differing de…
SchedulingTACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning
The surge of artificial intelligence, particularly large language models, has driven the rapid development of large-scale machine learning clusters. Executing distributed models on these clusters is often constrained by …