Load Balancing for AI Training Workloads
The extreme bandwidth demands of AI training has made load-balancing a critical component in AI fabrics, and a variety of load-balancing designs have emerged in recent work from both industry and research. However, there is currently little consensus on which design approach dominates or the conditions under which an approach dominates. We also lack an understanding of how far these approaches are from optimal. We provide a technical foundation for answering these questions by systematically evaluating leading load-balancing designs, while decoupling them from specific congestion control and loss recovery stacks. We find that load-balancing based on packet spraying dominates traditional approaches that load balance traffic at flow, flowlet, or subflow granularities. When comparing host- vs switch-based approaches to packet spraying, we find that they perform similarly in failure-free scenarios but that a host-based approach dominates under link failure because of its rapid visibility into end-to-end path conditions. We also identify that no leading approach achieves optimal O(1) queue scaling at maximum utilization. We demonstrate why a destination-based rotation (DR) discipline can reach this optimum and introduce Ofan, a switch-based implementation of DR that we show offers valuable performance gains over other packet spraying approaches.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Meta-Reinforcement Learning with Discrete World Models for Adaptive Load Balancing
We integrate a meta-reinforcement learning algorithm with the DreamerV3 architecture to improve load balancing in operating systems. This approach enables rapid adaptation to dynamic workloads with minimal retraining, ou…
ManagementMeta Reinforcement Learningreinforcement-learningReinforcement LearningSemi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments
Machine learning (ML) models are increasingly trained in clusters with non-dedicated workers possessing heterogeneous resources. In such scenarios, model training efficiency can be negatively affected by stragglers -- wo…
CPUGPUReinforcement Learning-Based Adaptive Load Balancing for Dynamic Cloud Environments
Efficient load balancing is crucial in cloud computing environments to ensure optimal resource utilization, minimize response times, and prevent server overload. Traditional load balancing algorithms, such as round-robin…
Cloud Computingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Revisiting Parameter Server in LLM Post-Training
Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced workload assumption no longer holds in larg…
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
Load imbalance is a long-standing challenge in Mixture-of-Experts (MoE) training and is exacerbated in reinforcement learning (RL) for LLMs, where hot experts can shift frequently across micro-batches. Existing MoE train…
Reinforcement Learning