TrainMover: An Interruption-Resilient and Reliable ML Training Runtime
Large-scale ML training jobs are frequently interrupted by hardware and software anomalies, failures, and management events. Existing solutions like checkpointing or runtime reconfiguration suffer from long downtimes, degraded performance, or undesired changes to training strategies. We present TrainMover, a resilient runtime that leverages standby machines to handle interruptions with minimal downtime and zero memory overhead. To achieve these goals, TrainMover introduces two key techniques: two-phase, delta-based communication group setups and communication-free sandboxed shadow iterations. Our evaluation shows that TrainMover consistently achieves second-level downtime across all evaluated models during migration, maintaining 99\% training efficiency during periodic 10-minute rebalancing. We also demonstrate the effectiveness of TrainMover in handling various interruptions.
Code (0)
등록된 구현이 없습니다.
Tasks
ManagementSchedulingSimilar Papers 제목 키워드 기반
Resilient Time-Varying Output Formation Tracking of Linear Multi-Agent Systems Against Unbounded FDI Sensor Attacks and Unreliable Digraphs
One salient feature of cooperative formation tracking is its distributed nature that relies on localized control and information sharing over a sparse communication network. That is, a distributed control manner could be…
Resilient Microgrid Formation Considering Communication Interruptions
Distribution system (DS) communication failures following extreme events often degrade monitoring and control functions, thus preventing the acquisition of complete global DS component state information, on which existin…
CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors
This work proposes the first strategy to make distributed training of neural networks resilient to computing errors, a problem that has remained unsolved despite being first posed in 1956 by von Neumann. He also speculat…
Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity
Federated Learning is a novel paradigm that involves learning from data samples distributed across a large network of clients while the data remains local. It is, however, known that federated learning is prone to multip…
Federated LearningResilient Identification of Distribution Network Topology
Network topology identification (TI) is an essential function for distributed energy resources management systems (DERMS) to organize and operate widespread distributed energy resources (DERs). In this paper, discriminan…
Management