paper-with-me

홈 › Papers

Distributed Training and Optimization Of Neural Networks

2020-12-03 · Jean-Roch Vlimant, Junqi Yin

Deep learning models are yielding increasingly better performances thanks to multiple factors. To be successful, model may have large number of parameters or complex architectures and be trained on large dataset. This leads to large requirements on computing resource and turn around time, even more so when hyper-parameter optimization is done (e.g search over model architectures). While this is a challenge that goes beyond particle physics, we review the various ways to do the necessary computations in parallel, and put it in the context of high energy physics.

📄 PDF Abstract BibTeX arXiv:2012.01839

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

A Comprehensive Survey on Distributed Training of Graph Neural Networks

2022-11-10 · Haiyang Lin, Mingyu Yan, Xiaochun Ye, Dongrui Fan 외

Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growin…

Survey

dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training

2022-05-05 · Hanpeng Hu, Chenyu Jiang, Yuchen Zhong, Yanghua Peng 외

Distributed training using multiple devices (e.g., GPUs) has been widely adopted for learning DNN models over large datasets. However, the performance of large-scale distributed training tends to be far from linear speed…

Characterizing and Understanding Distributed GNN Training on GPUs

2022-04-18 · Haiyang Lin, Mingyu Yan, Xiaocheng Yang, Mo Zou 외

Graph neural network (GNN) has been demonstrated to be a powerful model in many domains for its effectiveness in learning over graphs. To scale GNN training for large graphs, a widely adopted approach is distributed trai…

Graph Neural Network

Communication optimization strategies for distributed deep neural network training: A survey

2020-03-06 · Shuo Ouyang, Dezun Dong, Yemao Xu, Liquan Xiao

Recent trends in high-performance computing and deep learning have led to the proliferation of studies on large-scale deep neural network training. However, the frequent communication requirements among computation nodes…

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

2024-03-12 · Yunze Wei, Tianshuo Hu, Cong Liang, Yong Cui

The past few years have witnessed the flourishing of large-scale deep neural network models with ever-growing parameter numbers. Training such large-scale models typically requires massive memory and computing resources,…

GPU