Comprehensive Evaluation for a Large Scale Knowledge Graph Question Answering Service
Question answering systems for knowledge graph (KGQA), answer factoid questions based on the data in the knowledge graph. KGQA systems are complex because the system has to understand the relations and entities in the knowledge-seeking natural language queries and map them to structured queries against the KG to answer them. In this paper, we introduce Chronos, a comprehensive evaluation framework for KGQA at industry scale. It is designed to evaluate such a multi-component system comprehensively, focusing on (1) end-to-end and component-level metrics, (2) scalable to diverse datasets and (3) a scalable approach to measure the performance of the system prior to release. In this paper, we discuss the unique challenges associated with evaluating KGQA systems at industry scale, review the design of Chronos, and how it addresses these challenges. We will demonstrate how it provides a base for data-driven decisions and discuss the challenges of using it to measure and improve a real-world KGQA system.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Question AnsweringNatural Language QueriesQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PubGraph: A Large-Scale Scientific Knowledge Graph
Research publications are the primary vehicle for sharing scientific progress in the form of new discoveries, methods, techniques, and insights. Unfortunately, the lack of a large-scale, comprehensive, and easy-to-use re…
Community DetectionGraph EmbeddingInductive LearningKnowledge Graph Completion+2UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models
Model editing aims to enhance the accuracy and reliability of large language models (LLMs) by efficiently adjusting their internal parameters. Currently, most LLM editing datasets are confined to narrow knowledge domains…
Diversityknowledge editingKnowledge GraphsModel EditingOn Large-scale Evaluation of Embedding Models for Knowledge Graph Completion
Knowledge graph embedding (KGE) models are extensively studied for knowledge graph completion, yet their evaluation remains constrained by unrealistic benchmarks. Standard evaluation metrics rely on the closed-world assu…
Graph EmbeddingKnowledge Graph CompletionKnowledge Graph EmbeddingLink Prediction+2TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs
Multi-relational temporal graphs are powerful tools for modeling real-world data, capturing the evolving and interconnected nature of entities over time. Recently, many novel models are proposed for ML on such graphs int…
BenchmarkingKnowledge GraphsAFEC: A Knowledge Graph Capturing Social Intelligence in Casual Conversations
This paper introduces AFEC, an automatically curated knowledge graph based on people's day-to-day casual conversations. The knowledge captured in this graph bears potential for conversational systems to understand how pe…
ChatbotDiversityRetrieval