Benchmarking Machine Reading Comprehension: A Psychological Perspective
Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingMachine Reading ComprehensionNatural Language UnderstandingPositionReading ComprehensionSimilar Papers 제목 키워드 기반
SciMRC: Multi-perspective Scientific Machine Reading Comprehension
Scientific machine reading comprehension (SMRC) aims to understand scientific texts through interactions with humans by given questions. As far as we know, there is only one dataset focused on exploring full-text scienti…
Machine Reading ComprehensionReading ComprehensionMC\^2: Multi-perspective Convolutional Cube for Conversational Machine Reading Comprehension
Conversational machine reading comprehension (CMRC) extends traditional single-turn machine reading comprehension (MRC) by multi-turn interactions, which requires machines to consider the history of conversation. Most of…
Conversational Question AnsweringMachine Reading ComprehensionQuestion AnsweringReading ComprehensionUsing Psychological Characteristics of Situations for Social Situation Comprehension in Support Agents
Support agents that help users in their daily lives need to take into account not only the user's characteristics, but also the social situation of the user. Existing work on including social context uses some type of si…
ManagementBi-directional CognitiveThinking Network for Machine Reading Comprehension
We propose a novel Bi-directional Cognitive Knowledge Framework (BCKF) for reading comprehension from the perspective of complementary learning systems theory. It aims to simulate two ways of thinking in the brain to ans…
Machine Reading ComprehensionReading ComprehensionBi-directional Cognitive Thinking Network for Machine Reading Comprehension
We propose a novel Bi-directional Cognitive Knowledge Framework (BCKF) for reading comprehension from the perspective of complementary learning systems theory. It aims to simulate two ways of thinking in the brain to ans…
Machine Reading ComprehensionReading Comprehension