To Test Machine Comprehension, Start by Defining Comprehension
Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension -- a "Template of Understanding" -- for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine Reading ComprehensionReading ComprehensionSimilar Papers 제목 키워드 기반
Reading Comprehension Ability Test-A Turing Test for Reading Comprehension
Reading comprehension is an important ability of human intelligence. Literacy and numeracy are two most essential foundation for people to succeed at study, at work and in life. Reading comprehension ability is a core co…
Reading ComprehensionXCMRC: Evaluating Cross-lingual Machine Reading Comprehension
We present XCMRC, the first public cross-lingual language understanding (XLU) benchmark which aims to test machines on their cross-lingual reading comprehension ability. To be specific, XCMRC is a Cross-lingual Cloze-sty…
Machine Reading ComprehensionReading ComprehensionSentenceDataset for the First Evaluation on Chinese Machine Reading Comprehension
Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, existing reading comprehension datasets are mostly in English. To add diversity in reading compreh…
Chinese Reading ComprehensionDiversityMachine Reading ComprehensionReading ComprehensionOne-Topic-Doesn't-Fit-All: Transcreating Reading Comprehension Test for Personalized Learning
Personalized learning has gained attention in English as a Foreign Language (EFL) education, where engagement and motivation play crucial roles in reading comprehension. We propose a novel approach to generating personal…
Reading ComprehensionNatural Language Comprehension with the EpiReader
We present the EpiReader, a novel model for machine comprehension of text. Machine comprehension of unstructured, real-world text is a major research goal for natural language processing. Current tests of machine compreh…
Question AnsweringReading Comprehension