Survey on Large Scale Neural Network Training
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU device or can be trained using only a small per-GPU batch size. This survey provides a systematic overview of the approaches that enable more efficient DNNs training. We analyze techniques that save memory and make good use of computation and communication resources on architectures with a single or several GPUs. We summarize the main categories of strategies and compare strategies within and across categories. Along with approaches proposed in the literature, we discuss available implementations.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUSurveySimilar Papers 제목 키워드 기반
Telephone Surveys Meet Conversational AI: Evaluating a LLM-Based Telephone Survey System at Scale
Telephone surveys remain a valuable tool for gathering insights but typically require substantial resources in training and coordinating human interviewers. This work presents an AI-driven telephone survey system integra…
AI AgentLarge Language ModelSpeech-to-TextSurvey+2Deep Learning at Scale for the Construction of Galaxy Catalogs in the Dark Energy Survey
The scale of ongoing and future electromagnetic surveys pose formidable challenges to classify astronomical objects. Pioneering efforts on this front include citizen science campaigns adopted by the Sloan Digital Sky Sur…
ClusteringSurveyLarge-Scale Deep Learning Optimizations: A Comprehensive Survey
Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend longer training time on more computation …
Deep LearningSurveyResource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey
With rapidly increasing distributed deep learning workloads in large-scale data centers, efficient distributed deep learning framework strategies for resource allocation and workload scheduling have become the key to hig…
Deep LearningSchedulingSurveyConsistently Large Cosmic Flows on Scales of 100 Mpc/h: a Challenge for the Standard LCDM Cosmology
Peculiar velocity surveys have non-uniform spatial distributions of tracers, so that the bulk flow estimated from them does not correspond to that of a simple volume such as a sphere. Thus bulk flow estimates are general…