DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning
Knowledge Graph Embedding (KGE) is a popular method for KG reasoning and training KGEs with higher dimension are usually preferred since they have better reasoning capability. However, high-dimensional KGEs pose huge challenges to storage and computing resources and are not suitable for resource-limited or time-constrained applications, for which faster and cheaper reasoning is necessary. To address this problem, we propose DualDE, a knowledge distillation method to build low-dimensional student KGE from pre-trained high-dimensional teacher KGE. DualDE considers the dual-influence between the teacher and the student. In DualDE, we propose a soft label evaluation mechanism to adaptively assign different soft label and hard label weights to different triples, and a two-stage distillation approach to improve the student's acceptance of the teacher. Our DualDE is general enough to be applied to various KGEs. Experimental results show that our method can successfully reduce the embedding parameters of a high-dimensional KGE by 7 times - 15 times and increase the inference speed by 2 times - 6 times while retaining a high performance. We also experimentally prove the effectiveness of our soft label evaluation mechanism and two-stage distillation approach via ablation study.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph EmbeddingKnowledge DistillationKnowledge Graph EmbeddingKnowledge Graph EmbeddingsLink PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distilling Word Embeddings: An Encoding Approach
Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restr…
Word EmbeddingsGraph-based Knowledge Distillation by Multi-head Attention Network
Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN.…
Inductive BiasKnowledge DistillationMulti-Task LearningTransfer LearningDistilling Holistic Knowledge with Graph Neural Networks
Knowledge Distillation (KD) aims at transferring knowledge from a larger well-optimized teacher network to a smaller learnable student network.Existing KD methods have mainly considered two types of knowledge, namely the…
Knowledge DistillationHIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks
Researchers have recently proposed plenty of heterogeneous graph neural networks (HGNNs) due to the ubiquity of heterogeneous graphs in both academic and industrial areas. Instead of pursuing a more powerful HGNN model, …
Knowledge DistillationVocal Bursts Intensity PredictionCascade Graph Neural Networks for RGB-D Salient Object Detection
In this paper, we study the problem of salient object detection (SOD) for RGB-D images using both color and depth information.A major technical challenge in performing salient object detection fromRGB-D images is how to …
Objectobject-detectionObject DetectionRGB-D Salient Object Detection+2