What Makes Reading Comprehension Questions Easier?
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.
Code (1)
Tasks
Machine Reading ComprehensionMultiple-choiceReading ComprehensionSentenceSimilar Papers 제목 키워드 기반
How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading
Using questions in written text is an effective strategy to enhance readability. However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human read…
ArticlesMemorizationReading ComprehensionWhat Makes Reading Comprehension Questions Difficult? Investigating Variation in Passage Sources and Question Types
In order for a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems…
Logical ReasoningMultiple-choiceNatural Language UnderstandingReading ComprehensionQuestion Generation for Reading Comprehension Assessment by Modeling How and What to Ask
Reading is integral to everyday life, and yet learning to read is a struggle for many young learners. During lessons, teachers can use comprehension questions to increase engagement, test reading skills, and improve rete…
Question GenerationQuestion-GenerationReading ComprehensionQuestion Generation for Reading Comprehension Assessment by Modeling How and What to Ask
Reading is integral to everyday life, and yet learning to read is a struggle for many young learners. During lessons, teachers can use comprehension questions to increase engagement, test reading skills, and improve ret…
Question GenerationQuestion-GenerationReading ComprehensionWhat Makes Reading Comprehension Questions Difficult?
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However…
Logical ReasoningMultiple-choiceNatural Language UnderstandingReading Comprehension