Reducing Oversmoothing through Informed Weight Initialization in Graph Neural Networks
In this work, we generalize the ideas of Kaiming initialization to Graph Neural Networks (GNNs) and propose a new scheme (G-Init) that reduces oversmoothing, leading to very good results in node and graph classification tasks. GNNs are commonly initialized using methods designed for other types of Neural Networks, overlooking the underlying graph topology. We analyze theoretically the variance of signals flowing forward and gradients flowing backward in the class of convolutional GNNs. We then simplify our analysis to the case of the GCN and propose a new initialization method. Our results indicate that the new method (G-Init) reduces oversmoothing in deep GNNs, facilitating their effective use. Experimental validation supports our theoretical findings, demonstrating the advantages of deep networks in scenarios with no feature information for unlabeled nodes (i.e., ``cold start'' scenario).
Code (0)
등록된 구현이 없습니다.
Tasks
Graph ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations
This article derives and validates three principles for initialization and architecture selection in finite width graph neural networks (GNNs) with ReLU activations. First, we theoretically derive what is essentially the…
Analyzing the Effect of Embedding Norms and Singular Values to Oversmoothing in Graph Neural Networks
In this paper, we study the factors that contribute to the effect of oversmoothing in deep Graph Neural Networks (GNNs). Specifically, our analysis is based on a new metric (Mean Average Squared Distance - $MASED$) to qu…
Node ClassificationBeyond ReLU: Bifurcation, Oversmoothing, and Topological Priors
Graph Neural Networks (GNNs) learn node representations through iterative network-based message-passing. While powerful, deep GNNs suffer from oversmoothing, where node features converge to a homogeneous, non-informative…
LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training
Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untra…
Partially Trained Graph Convolutional Networks Resist Oversmoothing
In this work we investigate an observation made by Kipf \& Welling, who suggested that untrained GCNs can generate meaningful node embeddings. In particular, we investigate the effect of training only a single layer of a…