Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions
When answering natural language questions over knowledge bases, missing facts, incomplete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA settings, it has not been studied for QA over knowledge bases (KBQA). We create GrailQAbility, a new benchmark KBQA dataset with unanswerability, by first identifying various forms of KB incompleteness that make questions unanswerable, and then systematically adapting GrailQA (a popular KBQA dataset with only answerable questions). Experimenting with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance even after suitable adaptation for unanswerable questions. In addition, these often detect unanswerability for wrong reasons and find specific forms of unanswerability particularly difficult to handle. This underscores the need for further research in making KBQA systems robust to unanswerability
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
RetinaQA: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questions
An essential requirement for a real-world Knowledge Base Question Answering (KBQA) system is the ability to detect the answerability of questions when generating logical forms. However, state-of-the-art KBQA models assum…
Knowledge Base Question AnsweringQuestion AnsweringRetrievalvalid+1Robust Few-shot Transfer Learning for Knowledge Base Question Answering with Unanswerable Questions
Real-world KBQA applications require models that are (1) robust -- e.g., can differentiate between answerable and unanswerable questions, and (2) low-resource -- do not require large training data. Towards this goal, we …
Knowledge Base Question AnsweringQuestion AnsweringTransfer LearningEvaluating the Knowledge Dependency of Questions
The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU,…
Multiple-choiceHarnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs
Conversational Information Retrieval (CIR) systems, while offering intuitive access to information, face a significant challenge: reliably handling unanswerable questions to prevent the generation of misleading or halluc…
Reinforcement LearningInformation RetrievalMulti-Task LearningResponse GenerationTowards a Better Metric for Evaluating Question Generation Systems
There has always been criticism for using $n$-gram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems. However, these metrics continue to remain popular and are recently bein…
Knowledge GraphsQuestion GenerationQuestion-Generation