What Makes Machine Reading Comprehension Questions Difficult? Investigating Variation in Passage Sources and Question Types
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select passages to collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty. Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for technical passages. These results suggest that when creating a new benchmark dataset, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority.
Code (0)
등록된 구현이 없습니다.
Tasks
Logical ReasoningMachine Reading ComprehensionMultiple-choiceNatural Language UnderstandingReading ComprehensionSimilar Papers 제목 키워드 기반
What Makes Reading Comprehension Questions Easier?
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investiga…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionSentenceHow to Engage Your Readers? Generating Guiding Questions to Promote Active Reading
Using questions in written text is an effective strategy to enhance readability. However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human read…
ArticlesMemorizationReading ComprehensionCosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos Q…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionOn Making Reading Comprehension More Comprehensive
Machine reading comprehension, the task of evaluating a machine{'}s ability to comprehend a passage of text, has seen a surge in popularity in recent years. There are many datasets that are targeted at reading comprehens…
Machine Reading ComprehensionQuestion AnsweringReading ComprehensionTORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading c…
Machine Reading ComprehensionQuestion AnsweringReading Comprehension