MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs
Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significant challenges in terms of execution performance. To tackle this, distributed-memory solutions such as partitioning the graph to concurrently train multiple replicas of GNNs are in practice. However, approaches requiring a partitioned graph usually suffer from communication overhead and load imbalance, even under optimal partitioning and communication strategies due to irregularities in the neighborhood minibatch sampling. This paper proposes practical trade-offs for improving the sampling and communication overheads for representation learning on distributed graphs (using popular GraphSAGE architecture) by developing a parameterized continuous prefetch and eviction scheme on top of the state-of-the-art Amazon DistDGL distributed GNN framework, demonstrating about 15-40% improvement in end-to-end training performance on the National Energy Research Scientific Computing Center's (NERSC) Perlmutter supercomputer for various OGB datasets.
Code (1)
Tasks
Representation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FedFetch: Faster Federated Learning with Adaptive Downstream Prefetching
Federated learning (FL) is a machine learning paradigm that facilitates massively distributed model training with end-user data on edge devices directed by a central server. However, the large number of heterogeneous cli…
Federated LearningRudder: Steering Prefetching in Distributed GNN Training using LLM Agents
Large-scale Graph Neural Networks (GNNs) are typically trained by sampling a vertex's neighbors to a fixed distance. Because large input graphs are distributed, training requires frequent irregular communication that sta…
RapidGNN: Energy and Communication-Efficient Distributed Training on Large-Scale Graph Neural Networks
Graph Neural Networks (GNNs) have become popular across a diverse set of tasks in exploring structural relationships between entities. However, due to the highly connected structure of the datasets, distributed training …
Massively Distributed Antenna Systems with Non-Ideal Optical Fiber Front-hauls: A Promising Technology for 6G Wireless Communication Systems
Employing massively distributed antennas brings radio access points (RAPs) closer to users, thus enables aggressive spectrum reuse that can bridge gaps between the scarce spectrum resource and extremely high connection d…
Distributed Learning in the Non-Convex World: From Batch to Streaming Data, and Beyond
Distributed learning has become a critical enabler of the massively connected world envisioned by many. This article discusses four key elements of scalable distributed processing and real-time intelligence --- problems,…