Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs
Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semisupervised node classification on graphs, by training a student MLP by knowledge distillation from a teacher graph neural network (GNN). While previous studies have focused mostly on training the student MLP by matching the output probability distributions between the teacher and student models during distillation, it has not been systematically studied how to inject the structural information in an explicit and interpretable manner. Inspired by GNNs that separate feature transformation $T$ and propagation $\Pi$, we re-frame the distillation process as making the student MLP learn both $T$ and $\Pi$. Although this can be achieved by applying the inverse propagation $\Pi^{-1}$ before distillation from the teacher, it still comes with a high computational cost from large matrix multiplications during training. To solve this problem, we propose Propagate & Distill (P&D), which propagates the output of the teacher before distillation, which can be interpreted as an approximate process of the inverse propagation. We demonstrate that P&D can readily improve the performance of the student MLP.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Neural NetworkKnowledge DistillationNode ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semi-supervised node classification on graphs, by training a student MLP by knowledge distillation (KD) from a teacher graph neural network (GNN)…
Graph LearningGraph Neural NetworkKnowledge DistillationNode ClassificationEnsemble Teaching for Hybrid Label Propagation
Label propagation aims to iteratively diffuse the label information from labeled examples to unlabeled examples over a similarity graph. Current label propagation algorithms cannot consistently yield satisfactory perform…
Learning How to Propagate Messages in Graph Neural Networks
This paper studies the problem of learning message propagation strategies for graph neural networks (GNNs). One of the challenges for graph neural networks is that of defining the propagation strategy. For instance, the …
Deoscillated Graph Collaborative Filtering
Collaborative Filtering (CF) signals are crucial for a Recommender System~(RS) model to learn user and item embeddings. High-order information can alleviate the cold-start issue of CF-based methods, which is modelled thr…
Collaborative FilteringRecommendation SystemsCascading Adversarial Bias from Injection to Distillation in Language Models
Model distillation has become essential for creating smaller, deployable language models that retain larger system capabilities. However, widespread deployment raises concerns about resilience to adversarial manipulation…
Bias DetectionCode GenerationData Poisoning