PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering
Existing work on Temporal Question Answering (TQA) has predominantly focused on questions anchored to specific timestamps or events (e.g. "Who was the US president in 1970?"). Little work has studied questions whose temporal context is relative to the present time (e.g. "Who was the previous US president?"). We refer to this problem as Present-Anchored Temporal QA (PATQA). PATQA poses unique challenges: (1) large language models (LLMs) may have outdated knowledge, (2) complex temporal relationships (e.g. 'before', 'previous') are hard to reason, (3) multi-hop reasoning may be required, and (4) the gold answers of benchmarks must be continuously updated. To address these challenges, we introduce the PAT-Questions benchmark, which includes single and multi-hop temporal questions. The answers in PAT-Questions can be automatically refreshed by re-running SPARQL queries on a knowledge graph, if available. We evaluate several state-of-the-art LLMs and a SOTA temporal reasoning model (TEMPREASON-T5) on PAT-Questions through direct prompting and retrieval-augmented generation (RAG). The results highlight the limitations of existing solutions in PATQA and motivate the need for new methods to improve PATQA reasoning capabilities.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringRAGRetrieval-augmented GenerationSimilar Papers 제목 키워드 기반
Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions
Large Language Models (LLMs), such as the GPT-4 and LLaMA families, have demonstrated considerable success across diverse tasks, including multiple-choice questions (MCQs). However, these models exhibit a positional bias…
Decision MakingMultiple-choiceDense Clinical Contrasts Enhance Medical Knowledge Updating in Large Language Models
Medical knowledge changes continually, making large language models vulnerable to relying on outdated yet clinically plausible information. We study whether the format of supervision affects medical knowledge updating un…
PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs
Knowledge updating in pre-trained Large Language Models (LLMs) remains an important challenge. While continual training provides a potential avenue for knowledge updating, it continues to present substantial technical di…
Data AugmentationOn the Limits of LLM-as-Judge for Scientific Novelty Assessment
LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its e…
Information Association for Language Model Updating by Mitigating LM-Logical Discrepancy
Large Language Models~(LLMs) struggle with providing current information due to the outdated pre-training data. Existing methods for updating LLMs, such as knowledge editing and continual fine-tuning, have significant dr…
Answer GenerationArticlesknowledge editingLanguage Modeling+3