E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.
Code (1)
Tasks
Entity EmbeddingsEntity LinkingLanguage ModellingQuestion AnsweringRelation ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Entity-aware Transformers for Entity Search
Pre-trained language models such as BERT have been a key ingredient to achieve state-of-the-art results on a variety of tasks in natural language processing and, more recently, also in information retrieval.Recent resear…
Entity EmbeddingsEntity RetrievalKnowledge GraphsLanguage Modelling+2Analyzing how BERT performs entity matching
State-of-the-art Entity Matching (EM) approaches rely on transformer architectures, such as BERT, for generating highly contex-tualized embeddings of terms. The embeddings are then used to predict whether pairs of entity…
Entity ResolutionSemantic SimilaritySemantic Textual SimilarityBERT-based Ranking for Biomedical Entity Normalization
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully appl…
Word EmbeddingsEntity-Assisted Language Models for Identifying Check-worthy Sentences
We propose a new uniform framework for text classification and ranking that can automate the process of identifying check-worthy sentences in political debates and speech transcripts. Our framework combines the semantic …
Entity EmbeddingsSentencetext-classificationText ClassificationZero-shot Entity Linking with Efficient Long Range Sequence Modeling
This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to e…
Entity LinkingPosition