Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing
Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Question AnsweringClinical KnowledgeKnowledge GraphsSimilar Papers 제목 키워드 기반
Enhancing Serendipity Recommendation System by Constructing Dynamic User Knowledge Graphs with Large Language Models
The feedback loop in industrial recommendation systems reinforces homogeneous content, creates filter bubble effects, and diminishes user satisfaction. Recently, large language models(LLMs) have demonstrated potential in…
Recommendation SystemsKnowledge GraphsSerendipity with Generative AI: Repurposing knowledge components during polycrisis with a Viable Systems Model approach
Organisations face polycrisis uncertainty yet overlook embedded knowledge. We show how generative AI can operate as a serendipity engine and knowledge transducer to discover, classify and mobilise reusable components (mo…
Leveraging Large Language Models for Causal Discovery: a Constraint-based, Argumentation-driven Approach
Causal discovery seeks to uncover causal relations from data, typically represented as causal graphs, and is essential for predicting the effects of interventions. While expert knowledge is required to construct principl…
Evaluating Large Language Models in Scientific Discovery
Large language models (LLMs) are increasingly applied to scientific research, yet prevailing science benchmarks probe decontextualized knowledge and overlook the iterative reasoning, hypothesis generation, and observatio…
Bursting Filter Bubble: Enhancing Serendipity Recommendations with Aligned Large Language Models
Recommender systems (RSs) often suffer from the feedback loop phenomenon, e.g., RSs are trained on data biased by their recommendations. This leads to the filter bubble effect that reinforces homogeneous content and redu…
Profile GenerationRecommendation SystemsWorld Knowledge