Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Previous literatures show that pre-trained masked language models (MLMs) such as BERT can achieve competitive factual knowledge extraction performance on some datasets, indicating that MLMs can potentially be a reliable knowledge source. In this paper, we conduct a rigorous study to explore the underlying predicting mechanisms of MLMs over different extraction paradigms. By investigating the behaviors of MLMs, we find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts. Furthermore, incorporating illustrative cases and external contexts improve knowledge prediction mainly due to entity type guidance and golden answer leakage. Our findings shed light on the underlying predicting mechanisms of MLMs, and strongly question the previous conclusion that current MLMs can potentially serve as reliable factual knowledge bases.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Probit Estimation of Urban Bases of Environmental Awareness: Evidence from Sylhet City, Bangladesh
This paper evaluates the significant factors contributing to environmental awareness among individuals living in the urban area of Sylhet, Bangladesh. Ordered Probit(OPM) estimation is applied on the value of ten measure…
KAMEL : Knowledge Analysis with Multitoken Entities in Language Models
Large language models (LMs) have been shown to capture large amounts of relational knowledge from the pre-training corpus. These models can be probed for this factual knowledge by using cloze-style prompts as demonstrat…
Knowledge GraphsProbing Language ModelsHierarchical Neural Network for Extracting Knowledgeable Snippets and Documents
In this study, we focus on extracting knowledgeable snippets and annotating knowledgeable documents from Web corpus, consisting of the documents from social media and We-media. Informally, knowledgeable snippets refer to…
Knowledge Base ConstructionMultimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning
Addressing the gap in understanding visual comprehension in Large Language Models (LLMs), we designed a challenge-response study, subjecting Google Bard and GPT-Vision to 64 visual tasks, spanning categories like "Visual…
Common Sense ReasoningOptical Character RecognitionVisual ReasoningKnowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) is a mainstream method for improving performance on knowledge-intensive tasks. However,current RAG systems often place too much emphasis on retrieved contexts. This can lead to relian…
counterfactualRAGRetrievalRetrieval-augmented Generation