ERNIE: Enhanced Representation through Knowledge Integration
We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.
Code (19)
Tasks
Chinese Named Entity RecognitionChinese Sentence Pair ClassificationChinese Sentiment AnalysisCloze TestNamed Entity RecognitionNamed Entity Recognition (NER)Natural Language InferenceQuestion AnsweringSemantic SimilaritySemantic Textual SimilaritySentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic co…
AttributePredictionReferring Expression ComprehensionSentence+1ERNIE: Enhanced Language Representation with Informative Entities
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP task…
Entity LinkingEntity TypingKnowledge GraphsLinguistic Acceptability+6What Has Been Enhanced in my Knowledge-Enhanced Language Model?
Pretrained language models (LMs) do not capture factual knowledge very well. This has led to the development of a number of knowledge integration (KI) methods which aim to incorporate external knowledge into pretrained L…
Graph AttentionLanguage ModelingLanguage ModellingPre-trained Language Model for Web-scale Retrieval in Baidu Search
Retrieval is a crucial stage in web search that identifies a small set of query-relevant candidates from a billion-scale corpus. Discovering more semantically-related candidates in the retrieval stage is very promising t…
Language ModelingLanguage ModellingRetrievalERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI
We examine the effect of domain-specific external knowledge variations on deep large scale language model performance. Recent work in enhancing BERT with external knowledge has been very popular, resulting in models such…
Language ModelingLanguage ModellingNatural Language Inference