Reducing Communication in Graph Neural Network Training
Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the naturally sparse connectivity information of the data. GNNs represent this connectivity as sparse matrices, which have lower arithmetic intensity and thus higher communication costs compared to dense matrices, making GNNs harder to scale to high concurrencies than convolutional or fully-connected neural networks. We introduce a family of parallel algorithms for training GNNs and show that they can asymptotically reduce communication compared to previous parallel GNN training methods. We implement these algorithms, which are based on 1D, 1.5D, 2D, and 3D sparse-dense matrix multiplication, using torch.distributed on GPU-equipped clusters. Our algorithms optimize communication across the full GNN training pipeline. We train GNNs on over a hundred GPUs on multiple datasets, including a protein network with over a billion edges.
Code (2)
Tasks
GPUGraph Neural NetworkSimilar Papers 제목 키워드 기반
Distributed Training of Large Graph Neural Networks with Variable Communication Rates
Training Graph Neural Networks (GNNs) on large graphs presents unique challenges due to the large memory and computing requirements. Distributed GNN training, where the graph is partitioned across multiple machines, is a…
graph partitioningDGCL: an efficient communication library for distributed GNN training
Graph neural networks (GNNs) have gained increasing popularity in many areas such as e-commerce, social networks and bio-informatics. Distributed GNN training is essential for handling large graphs and reducing the execu…
GPUFedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networks
Methods for training models on graphs distributed across multiple clients have recently grown in popularity, due to the size of these graphs as well as regulations on keeping data where it is generated. However, the cros…
Federated LearningNode ClassificationCommunication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsification
Federated Knowledge Graphs Embedding learning (FKGE) encounters challenges in communication efficiency stemming from the considerable size of parameters and extensive communication rounds. However, existing FKGE methods …
Entity EmbeddingsGraph EmbeddingKnowledge Graph EmbeddingKnowledge GraphsFederated Graph Learning with Adaptive Importance-based Sampling
For privacy-preserving graph learning tasks involving distributed graph datasets, federated learning (FL)-based GCN (FedGCN) training is required. A key challenge for FedGCN is scaling to large-scale graphs, which typica…
Federated LearningGraph LearningGraph SamplingPrivacy Preserving