Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension
Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question given a contextual paragraph. However, for many applications of multiple-choice MRC systems there are two additional considerations. For multiple-choice exams there is often a negative marking scheme; there is a penalty for an incorrect answer. This means that the system is required to have an idea of the uncertainty in the predicted answer. The second consideration is that many multiple-choice questions have the option of none of the above (NOA) indicating that none of the answers is applicable, rather than there always being the correct answer in the list of choices. This paper investigates both of these issues by making use of predictive uncertainty. It is shown that uncertainty does allow questions that the system is not confident about to be detected. Additionally we show that uncertainty outperforms a system explicitly built with an NOA option for the ReClor corpus.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine Reading ComprehensionMultiple-choiceReading ComprehensionSimilar Papers 제목 키워드 기반
Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension
Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question giv…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionCLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering
Recent Vision-Language Models (VLMs) have demonstrated remarkable capabilities in visual understanding and reasoning, and in particular on multiple-choice Visual Question Answering (VQA). Still, these models can make dis…
Multiple-choiceQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions
When answering natural language questions over knowledge bases, missing facts, incomplete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA se…
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above
Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA's format, as it struggles to: 1) test gener…
AllMultiple-choiceMultiple Choice Question Answering (MCQA)Question AnsweringSQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0
Existing machine reading comprehension models are reported to be brittle for adversarially perturbed questions when optimizing only for accuracy, which led to the creation of new reading comprehension benchmarks, such as…
Machine Reading ComprehensionReading Comprehension