Combining Contrastive Learning and Knowledge Graph Embeddings to develop medical word embeddings for the Italian language
Word embeddings play a significant role in today's Natural Language Processing tasks and applications. While pre-trained models may be directly employed and integrated into existing pipelines, they are often fine-tuned to better fit with specific languages or domains. In this paper, we attempt to improve available embeddings in the uncovered niche of the Italian medical domain through the combination of Contrastive Learning (CL) and Knowledge Graph Embedding (KGE). The main objective is to improve the accuracy of semantic similarity between medical terms, which is also used as an evaluation task. Since the Italian language lacks medical texts and controlled vocabularies, we have developed a specific solution by combining preexisting CL methods (multi-similarity loss, contextualization, dynamic sampling) and the integration of KGEs, creating a new variant of the loss. Although without having outperformed the state-of-the-art, represented by multilingual models, the obtained results are encouraging, providing a significant leap in performance compared to the starting model, while using a significantly lower amount of data.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningGraph EmbeddingKnowledge Graph EmbeddingKnowledge Graph EmbeddingsSemantic SimilaritySemantic Textual SimilarityWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion
Knowledge Graph Embeddings (KGE) aim to map entities and relations to low dimensional spaces and have become the \textit{de-facto} standard for knowledge graph completion. Most existing KGE methods suffer from the sparsi…
Contrastive LearningKnowledge Graph CompletionKnowledge Graph EmbeddingsKnowledge GraphsAdversarial Contrastive Estimation
Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in …
Contrastive LearningKnowledge Graph EmbeddingsKnowledge GraphsLearning Word Embeddings+1Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs
Biomedical Knowledge Graphs (BKGs) integrate diverse datasets to elucidate complex relationships within the biomedical field. Effective link prediction on these graphs can uncover valuable connections, such as potential …
Contrastive LearningGraph EmbeddingKnowledge Graph EmbeddingKnowledge Graphs+2Temporal Graph Representation Learning with Adaptive Augmentation Contrastive
Temporal graph representation learning aims to generate low-dimensional dynamic node embeddings to capture temporal information as well as structural and property information. Current representation learning methods for …
Contrastive LearningGraph Representation LearningRepresentation LearningBIG MOOD: Relating Transformers to Explicit Commonsense Knowledge
We introduce a simple yet effective method of integrating contextual embeddings with commonsense graph embeddings, dubbed BERT Infused Graphs: Matching Over Other embeDdings. First, we introduce a preprocessing method to…
Language ModelingLanguage Modelling