CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors
While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradigms predominantly rely on lexical-level similarity or manual heuristics, often lacking sufficient data-driven validation. We address this by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus, to distill six key personalization factors as evaluative dimensions. Accordingly, we introduce CoPA, a benchmark with 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA provides a more comprehensive and discriminative standard for evaluating personalized QA than generic metrics. The code is available at https://github.com/bjzgcai/CoPA.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items
LLMs can now perform a variety of complex writing tasks. They also excel in answering questions pertaining to natural language inference and commonsense reasoning. Composing these questions is itself a skilled writing ta…
Natural Language InferenceCommonsense Knowledge-Augmented Pretrained Language Models for Causal Reasoning Classification
Commonsense knowledge can be leveraged for identifying causal relations in text. In this work, we convert triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, to natural language text and continu…
ClassificationCommonsense Causal ReasoningLanguage ModelingLanguage ModellingEgo-Grounding for Personalized Question-Answering in Egocentric Videos
We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this …
Enhancing Multiple-Choice Question Answering with Causal Knowledge
The task of causal question answering aims to reason about causes and effects over a provided real or hypothetical premise. Recent approaches have converged on using transformer-based language models to solve question an…
Multiple-choiceQuestion AnsweringLaMP-QA: A Benchmark for Personalized Long-form Question Answering
Personalization is essential for question answering systems that are user-centric. Despite its importance, personalization in answer generation has been relatively underexplored. This is mainly due to lack of resources f…
Answer GenerationFormLong Form Question AnsweringQuestion Answering