Knowledge Base Embedding By Cooperative Knowledge Distillation

2020-12-01 · COLING 2020 8 · Rapha{\"e}l Sourty, Jose G. Moreno, Fran{\c{c}}ois-Paul Servant, Lynda Tamine-Lechani

Knowledge bases are increasingly exploited as gold standard data sources which benefit various knowledge-driven NLP tasks. In this paper, we explore a new research direction to perform knowledge base (KB) representation learning grounded with the recent theoretical framework of knowledge distillation over neural networks. Given a set of KBs, our proposed approach KD-MKB, learns KB embeddings by mutually and jointly distilling knowledge within a dynamic teacher-student setting. Experimental results on two standard datasets show that knowledge distillation between KBs through entity and relation inference is actually observed. We also show that cooperative learning significantly outperforms the two proposed baselines, namely traditional and sequential distillation.

📄 PDF Abstract BibTeX

Code (1)

raphaelsty/mkb 공식 구현 pytorch

Tasks

Knowledge Distillation Representation Learning

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Cooperative Knowledge Distillation: A Learner Agnostic Approach

2024-02-02 · Michael Livanos, Ian Davidson, Stephen Wong

Knowledge distillation is a simple but powerful way to transfer knowledge between a teacher model to a student model. Existing work suffers from at least one of the following key limitations in terms of direction and sco…

counterfactual Knowledge Distillation Self-Supervised Learning Transfer Learning

CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers

2024-04-09 · Lakshmi Nair

Contrastive Language-Image Pre-training (CLIP) has been shown to improve zero-shot generalization capabilities of language and vision models. In this paper, we extend CLIP for efficient knowledge distillation, by utilizi…

Knowledge Distillation Zero-shot Generalization

Learning from Diverse Reasoning Paths with Routing and Collaboration

2025-08-23 · Zhenyu Lei, Zhen Tan, Song Wang, Yaochen Zhu 외 arxiv

Advances in large language models (LLMs) significantly enhance reasoning capabilities but their deployment is restricted in resource-constrained scenarios. Knowledge distillation addresses this by transferring knowledge …

Knowledge Distillation

DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings

2021-12-10 · Chaochen Gao, Xing Wu, Peng Wang, Jue Wang 외

Large-scale contrastive learning models can learn very informative sentence embeddings, but are hard to serve online due to the huge model size. Therefore, they often play the role of "teacher", transferring abilities to…

Contrastive Learning Knowledge Distillation Semantic Textual Similarity Sentence+4

Jointly Learning Knowledge Embedding and Neighborhood Consensus with Relational Knowledge Distillation for Entity Alignment

2022-01-25 · Xinhang Li, Yong Zhang, Chunxiao Xing

Entity alignment aims at integrating heterogeneous knowledge from different knowledge graphs. Recent studies employ embedding-based methods by first learning the representation of Knowledge Graphs and then performing ent…

Benchmarking Entity Alignment Entity Embeddings Knowledge Distillation+2