Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?
Estimating the cognitive complexity of reading comprehension (RC) items is crucial for assessing item difficulty before it is administered to learners. Unlike syntactic and semantic features, such as passage length or semantic similarity between options, cognitive features that arise during answer reasoning are not readily extractable using existing NLP tools and have traditionally relied on human annotation. In this study, we examine whether large language models (LLMs) can estimate the cognitive complexity of RC items by focusing on two dimensions-Evidence Scope and Transformation Level-that indicate the degree of cognitive burden involved in reasoning about the answer. Our experimental results demonstrate that LLMs can approximate the cognitive complexity of items, indicating their potential as tools for prior difficulty analysis. Further analysis reveals a gap between LLMs' reasoning ability and their metacognitive awareness: even when they produce correct answers, they sometimes fail to correctly identify the features underlying their own reasoning process.
Code (0)
등록된 구현이 없습니다.
Tasks
Reading ComprehensionSemantic SimilaritySimilar Papers 제목 키워드 기반
ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities
Large language models (LLMs) have demonstrated potential in educational applications, yet their capacity to accurately assess the cognitive alignment of reading materials with students' developmental stages remains insuf…
Reading ComprehensionZero-Shot LearningGenerating Reading Comprehension Exercises with Large Language Models for Educational Applications
With the rapid development of large language models (LLMs), the applications of LLMs have grown substantially. In the education domain, LLMs demonstrate significant potential, particularly in automatic text generation, w…
Reading ComprehensionText GenerationTowards a Better Understanding Human Reading Comprehension with Brain Signals
Reading comprehension is a complex cognitive process involving many human brain activities. Plenty of works have studied the patterns and attention allocations of reading comprehension in information retrieval related sc…
EEGElectroencephalogram (EEG)Information RetrievalReading Comprehension+3Integrating LLM, EEG, and Eye-Tracking Biomarker Analysis for Word-Level Neural State Classification in Semantic Inference Reading Comprehension
With the recent proliferation of large language models (LLMs), such as Generative Pre-trained Transformers (GPT), there has been a significant shift in exploring human and machine comprehension of semantic language meani…
EEGFeature EngineeringReading ComprehensionGraph Representations for Reading Comprehension Analysis using Large Language Model and Eye-Tracking Biomarker
Reading comprehension is a fundamental skill in human cognitive development. With the advancement of Large Language Models (LLMs), there is a growing need to compare how humans and LLMs understand language across differe…
Emotion InterpretationReading ComprehensionInformation Retrieval