Static Embeddings as Efficient Knowledge Bases?
Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge
There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense re…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)CoLAKE: Contextualized Language and Knowledge Embedding
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the per…
Entity EmbeddingsKnowledge Graph CompletionKnowledge ProbingA Birds Eye View on Knowledge Graph Embeddings, Software Libraries, Applications and Challenges
In recent years, Knowledge Graph (KG) development has attracted significant researches considering the applications in web search, relation prediction, natural language processing, information retrieval, question answeri…
Information RetrievalKnowledge Graph CompletionKnowledge Graph EmbeddingsKnowledge Graphs+4Joint Semantic and Distributional Word Representations with Multi-Graph Embeddings
Word embeddings continue to be of great use for NLP researchers and practitioners due to their training speed and easiness of use and distribution. Prior work has shown that the representation of those words can be impro…
Word EmbeddingsComing to its senses: Lessons learned from Approximating Retrofitted BERT representations for Word Sense information
Retrofitting static vector space word representations using external knowledge bases has yielded substantial improvements in their lexical-semantic capacities but is non-trivial to apply to contextual word embeddings (CW…
Word Embeddings