Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningManagementSchedulingSimilar Papers 제목 키워드 기반
Rethinking Spatio-Temporal Anomaly Detection: A Vision for Causality-Driven Cybersecurity
As cyber-physical systems grow increasingly interconnected and spatially distributed, ensuring their resilience against evolving cyberattacks has become a critical priority. Spatio-Temporal Anomaly detection plays an imp…
Anomaly DetectionGraph LearningSpatiotemporal Synchronization of Distributed Arrays using Particle-Based Loopy Belief Propagation
Sensing and imaging with distributed radio infrastructures (e.g., distributed MIMO, wireless sensor networks, multistatic radar) rely on knowledge of the positions, orientations, and clock parameters of distributed apert…
SpiNNTools: The Execution Engine for the SpiNNaker Platform
Distributed systems are becoming more common place, as computers typically contain multiple computation processors. The SpiNNaker architecture is such a distributed architecture, containing millions of cores connected wi…
RetrievalScalable high-dimensional indexing and searching with Hadoop
While high-dimensional search-by-similarity techniques reached their maturity and in overall provide good performance, most of them are unable to cope with very large multimedia collections. The 'big data' challenge howe…
Vocal Bursts Intensity PredictionCommunication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey
With the rapid growth in the volume of data sets, models, and devices in the domain of deep learning, there is increasing attention on large-scale distributed deep learning. In contrast to traditional distributed deep le…
Data CompressionDeep LearningSchedulingSurvey