Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose single-token name is found in common LM vocabularies. Furthermore, the main benefit of this paradigm, namely querying the KB using a variety of natural language paraphrases, is underexplored so far. Here, we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.
Code (1)
Tasks
World KnowledgeSimilar Papers 제목 키워드 기반
Linking Graph Entities with Multiplicity and Provenance
Entity linking and resolution is a fundamental database problem with applications in data integration, data cleansing, information retrieval, knowledge fusion, and knowledge-base population. It is the task of accurately …
AttributeData IntegrationEntity LinkingInformation Retrieval+3Improving Zero-Shot Multi-Lingual Entity Linking
Entity linking -- the task of identifying references in free text to relevant knowledge base representations -- often focuses on single languages. We consider multilingual entity linking, where a single model is trained …
Entity LinkingKnowledge Enhanced Contextual Word Representations
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a g…
Entity LinkingEntity TypingLanguage ModelingLanguage Modelling+3Learning Entity Representations for Few-Shot Reconstruction of Wikipedia Categories
Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation …
Entity TypingLanguage ModelingLanguage ModellingLearning Word Embeddings+2The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMs
Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their representations and justifies their use as knowledge bases. Previous work has focused on probing PLMs for factual knowledge by measuring how …
Retrieval