Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning
Though language model text embeddings have revolutionized NLP research, their ability to capture high-level semantic information, such as relations between entities in text, is limited. In this paper, we propose a novel contrastive learning framework that trains sentence embeddings to encode the relations in a graph structure. Given a sentence (unstructured text) and its graph, we use contrastive learning to impose relation-related structure on the token-level representations of the sentence obtained with a CharacterBERT (El Boukkouri et al.,2020) model. The resulting relation-aware sentence embeddings achieve state-of-the-art results on the relation extraction task using only a simple KNN classifier, thereby demonstrating the success of the proposed method. Additional visualization by a tSNE analysis shows the effectiveness of the learned representation space compared to baselines. Furthermore, we show that we can learn a different space for named entity recognition, again using a contrastive learning objective, and demonstrate how to successfully combine both representation spaces in an entity-relation task.
Code (1)
Tasks
Contrastive LearningLanguage ModelingLanguage Modellingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)RelationRelation ExtractionSentenceSentence EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification
Hierarchical multi-label text classification (HMTC) aims at utilizing a label hierarchy in multi-label classification. Recent approaches to HMTC deal with the problem of imposing an over-constrained premise on the output…
Contrastive LearningMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONMulti Label Text Classification+3Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions
Natural language definitions possess a recursive, self-explanatory semantic structure that can support representation learning methods able to preserve explicit conceptual relations and constraints in the latent space. T…
Learning Word EmbeddingsRepresentation LearningWord EmbeddingsContrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry
Recent trends in NLP utilize knowledge graphs (KGs) to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between docu…
Contrastive LearningDomain AdaptationKnowledge GraphsContrastive Loss is All You Need to Recover Analogies as Parallel Lines
While static word embedding models are known to represent linguistic analogies as parallel lines in high-dimensional space, the underlying mechanism as to why they result in such geometric structures remains obscure. We …
AllWord EmbeddingsAlignment and Outer Shell Isotropy for Hyperbolic Graph Contrastive Learning
Learning good self-supervised graph representations that are beneficial to downstream tasks is challenging. Among a variety of methods, contrastive learning enjoys competitive performance. The embeddings of contrastive l…
Contrastive LearningGraph EmbeddingSelf-Supervised Learning