Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. We solicit and verify questions and answers for this challenge through a 4-step crowdsourcing experiment. Our challenge dataset contains 6,500+ questions for 1000+ paragraphs across 7 different domains (elementary school science, news, travel guides, fiction stories, etc) bringing in linguistic diversity to the texts and to the questions wordings. On a subset of our dataset, we found human solvers to achieve an F1-score of 88.1{\%}. We analyze a range of baselines, including a recent state-of-art reading comprehension system, and demonstrate the difficulty of this challenge, despite a high human performance. The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityNatural Language InferenceQuestion AnsweringReading ComprehensionSentenceSimilar Papers 제목 키워드 기반
DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension
We present DREAM, the first dialogue-based multiple-choice reading comprehension dataset. Collected from English-as-a-foreign-language examinations designed by human experts to evaluate the comprehension level of Chinese…
Dialogue UnderstandingMultiple-choiceReading ComprehensionSentence+1DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
We present DREAM, the first dialogue-based multiple-choice reading comprehension data set. Collected from English as a Foreign Language examinations designed by human experts to evaluate the comprehension level of Chines…
Dialogue UnderstandingMultiple-choiceReading ComprehensionSentence+1On Making Reading Comprehension More Comprehensive
Machine reading comprehension, the task of evaluating a machine{'}s ability to comprehend a passage of text, has seen a surge in popularity in recent years. There are many datasets that are targeted at reading comprehens…
Machine Reading ComprehensionQuestion AnsweringReading ComprehensionLooking Beyond Short-Premise Natural Language Inference for Downstream Tasks
Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the mul…
Multiple-choiceNatural Language InferenceReading ComprehensionLooking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization
Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the mul…
Multiple-choiceNatural Language InferenceQuestion AnsweringReading Comprehension+2