Characterizing and Understanding Distributed GNN Training on GPUs
Graph neural network (GNN) has been demonstrated to be a powerful model in many domains for its effectiveness in learning over graphs. To scale GNN training for large graphs, a widely adopted approach is distributed training which accelerates training using multiple computing nodes. Maximizing the performance is essential, but the execution of distributed GNN training remains preliminarily understood. In this work, we provide an in-depth analysis of distributed GNN training on GPUs, revealing several significant observations and providing useful guidelines for both software optimization and hardware optimization.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Neural NetworkSimilar Papers 제목 키워드 기반
Characterizing and Understanding HGNN Training on GPUs
Owing to their remarkable representation capabilities for heterogeneous graph data, Heterogeneous Graph Neural Networks (HGNNs) have been widely adopted in many critical real-world domains such as recommendation systems …
GPURecommendation SystemsCharacterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
The rapid scaling of Large Language Models (LLMs) has pushed training workloads far beyond the limits of single-node analysis, demanding a deeper understanding of how these models behave across large-scale, multi-GPU sys…
Theano-MPI: a Theano-based Distributed Training Framework
We develop a scalable and extendable training framework that can utilize GPUs across nodes in a cluster and accelerate the training of deep learning models based on data parallelism. Both synchronous and asynchronous tra…
Deep LearningDD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), de…
Autonomous NavigationGPUNavigatePointGoal Navigation+3DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training
We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach, to reduce the memory consumption of contrastive loss when training contrastive learning models. Our approach decomposes the contrastive loss a…
Contrastive LearningGPU