What Makes Reading Comprehension Questions Difficult? Investigating Variation in Passage Sources and Question Types
In order for a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know what kinds of passages and their sources help us collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. We find that passage source, length, and readability measures do not significantly affect question difficulty. Among seven question types we manually annotate, questions that require numerical reasoning and logical reasoning are relatively difficult but their frequencies depend on the passage sources. These results suggest that when creating a new benchmark dataset, we do not have to use difficult passages but select passage sources carefully so that it has questions that involve linguistic phenomena we are interested in.
Code (0)
등록된 구현이 없습니다.
Tasks
Logical ReasoningMultiple-choiceNatural Language UnderstandingReading ComprehensionSimilar Papers 제목 키워드 기반
What Makes Reading Comprehension Questions Difficult?
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However…
Logical ReasoningMultiple-choiceNatural Language UnderstandingReading ComprehensionWhat Makes Machine Reading Comprehension Questions Difficult? Investigating Variation in Passage Sources and Question Types
For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However…
Logical ReasoningMachine Reading ComprehensionMultiple-choiceNatural Language Understanding+1How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading
Using questions in written text is an effective strategy to enhance readability. However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human read…
ArticlesMemorizationReading ComprehensionWhat Makes Reading Comprehension Questions Easier?
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investiga…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionSentenceDifficulty Controllable Generation of Reading Comprehension Questions
We investigate the difficulty levels of questions in reading comprehension datasets such as SQuAD, and propose a new question generation setting, named Difficulty-controllable Question Generation (DQG). Taking as input a…
Question GenerationQuestion-GenerationReading ComprehensionSentence