Evaluating the Knowledge Base Completion Potential of GPT
Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT's potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge Base CompletionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Evaluating Embedding Models for Knowledge Base Completion
Knowledge bases contribute to many web search and mining tasks, yet they are often incomplete. To add missing facts to a given knowledge base, various embedding models have been proposed in the recent literature. Perhaps…
Knowledge Base CompletionQuestion AnsweringEvaluating Language Models for Knowledge Base Completion
Structured knowledge bases (KBs) are a foundation of many intelligent applications, yet are notoriously incomplete. Language models (LMs) have recently been proposed for unsupervised knowledge base completion (KBC), yet,…
Knowledge Base CompletionA survey of embedding models of entities and relationships for knowledge graph completion
Knowledge graphs (KGs) of real-world facts about entities and their relationships are useful resources for a variety of natural language processing tasks. However, because knowledge graphs are typically incomplete, it is…
Knowledge Base CompletionKnowledge Graph CompletionKnowledge GraphsLink PredictionRevisiting Inferential Benchmarks for Knowledge Graph Completion
Knowledge Graph (KG) completion is the problem of extending an incomplete KG with missing facts. A key feature of Machine Learning approaches for KG completion is their ability to learn inference patterns, so that the pr…
Knowledge Graph CompletionAuto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation
The increasing adoption of foundation models as agents across diverse domains necessitates a robust evaluation framework. Current methods, such as LLM-as-a-Judge, focus only on final outputs, overlooking the step-by-step…