Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
SurveySimilar Papers 제목 키워드 기반
KLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships
Interactions between entities in knowledge graph (KG) provide rich knowledge for language representation learning. However, existing knowledge-enhanced pretrained language models (PLMs) only focus on entity information a…
Entity LinkingEntity TypingLanguage ModelingLanguage Modelling+4A Survey of Knowledge-Enhanced Text Generation
The goal of text generation is to make machines express in human language. It is one of the most important yet challenging tasks in natural language processing (NLP). Since 2014, various neural encoder-decoder models pio…
DecoderSurveyText GenerationA Survey of Knowledge Enhanced Pre-trained Models
Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) af…
Logical ReasoningRepresentation LearningSelf-Supervised LearningSurveyLarge Language Model Enhanced Knowledge Representation Learning: A Survey
The integration of Large Language Models (LLM) with Knowledge Representation Learning (KRL) signifies a significant advancement in the field of artificial intelligence (AI), enhancing the ability to capture and utilize b…
Language ModelingLanguage ModellingLarge Language ModelRepresentation Learning+1Commonsense Knowledge-Augmented Pretrained Language Models for Causal Reasoning Classification
Commonsense knowledge can be leveraged for identifying causal relations in text. In this work, we convert triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, to natural language text and continu…
ClassificationCommonsense Causal ReasoningLanguage ModelingLanguage Modelling