Papers Knowledge Probing
“Knowledge Probing” 태그가 달린 논문 46편 · 필터 해제
Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing
The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such ga…
Knowledge Probing"Let's Argue Both Sides": Argument Generation Can Force Small Models to Utilize Previously Inaccessible Reasoning Capabilities
Large Language Models (LLMs), despite achieving state-of-the-art results in a number of evaluation tasks, struggle to maintain their performance when logical reasoning is strictly required to correctly infer a prediction…
Knowledge ProbingLogical ReasoningCorrelation and Navigation in the Vocabulary Key Representation Space of Language Models
Language model (LM) decoding is based on the next-token prediction (NTP) probability distribution. For neural LMs (e.g., Transformer-based), NTP distribution is essentially a softmax-regularized dot product between an en…
DiversityKnowledge ProbingLM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models
Knowledge probing evaluates the extent to which a language model (LM) has acquired relational knowledge during its pre-training phase. It provides a cost-effective means of comparing LMs of different sizes and training s…
Continual LearningKnowledge ProbingLanguage ModelingLanguage ModellingKnowledge Probing for Graph Representation Learning
Graph learning methods have been extensively applied in diverse application areas. However, what kind of inherent graph properties e.g. graph proximity, graph structural information has been encoded into graph representa…
Graph ClassificationGraph LearningGraph Representation LearningKnowledge Probing+3What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models
Language models often struggle with handling factual knowledge, exhibiting factual hallucination issue. This makes it vital to evaluate the models' ability to recall its parametric knowledge about facts. In this study, w…
DecoderHallucinationIn-Context LearningKnowledge ProbingChaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies
This study explores the sycophantic tendencies of Large Language Models (LLMs), where these models tend to provide answers that match what users want to hear, even if they are not entirely correct. The motivation behind …
HallucinationKnowledge ProbingMisinformationBEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models
Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurati…
Factual probeGeneral KnowledgeKnowledge ProbingLanguage Modeling+2Unveiling LLMs: The Evolution of Latent Representations in a Dynamic Knowledge Graph
Large Language Models (LLMs) demonstrate an impressive capacity to recall a vast range of factual knowledge. However, understanding their underlying reasoning and internal mechanisms in exploiting this knowledge remains …
Claim VerificationCommon Sense ReasoningKnowledge GraphsKnowledge Probing+3TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models
KEPLMs are pre-trained models that utilize external knowledge to enhance language understanding. Previous language models facilitated knowledge acquisition by incorporating knowledge-related pre-training tasks learned fr…
Knowledge GraphsKnowledge ProbingTracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire a…
Cross-Lingual TransferKnowledge ProbingRepresentation LearningLearning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation
We evaluate the ability of Large Language Models (LLMs) to discern and express their internal knowledge state, a key factor in countering factual hallucination and ensuring reliable application of LLMs. We observe a robu…
HallucinationKnowledge Probingreinforcement-learningReinforcement LearningLanguage Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?
Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-res…
Knowledge ProbingRetrievalTransfer LearningKiki or Bouba? Sound Symbolism in Vision-and-Language Models
Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings …
Knowledge ProbingGive Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Pre-trained Language Models (PLMs) are trained on vast unlabeled data, rich in world knowledge. This fact has sparked the interest of the community in quantifying the amount of factual knowledge present in PLMs, as this …
Knowledge ProbingWorld KnowledgePromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
Biomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends. However, most current benchmarks: (a) are limited to English which m…
Dialogue GenerationDialogue UnderstandingKnowledge ProbingLanguage Modeling+5Assessing the Reliability of Large Language Model Knowledge
Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. LLMs are typically evaluated using accuracy, yet this metric does not capture the vulnerabilit…
HallucinationKnowledge ProbingLanguage ModelingLanguage Modelling+2Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata
In this work, we explore the use of Large Language Models (LLMs) for knowledge engineering tasks in the context of the ISWC 2023 LM-KBC Challenge. For this task, given subject and relation pairs sourced from Wikidata, we…
Knowledge ProbingPop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?
Recent breakthroughs in pre-trained code models, such as CodeBERT and Codex, have shown their superior performance in various downstream tasks. The correctness and unambiguity of API usage among these code models are cru…
Code GenerationKnowledge ProbingCan Language Models Solve Graph Problems in Natural Language?
Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures, such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense rea…
In-Context LearningKnowledge ProbingMulti-hop Question AnsweringQuestion Answering