Probing Difficulty and Discrimination of Natural Language Questions With Item Response Theory
Item Response Theory (IRT) has been extensively used to characterize question difficulty for human subjects in domains including cognitive psychology and education (Primi et al.,2014; Downing, 2003). In this work, we explore IRT to characterize the difficulty and discrimination of natural language questions in Question-Answering datasets. We use HotPotQA for illustration. Our analysis reveals significant variations along these traits, as well as interdependence between them. Additionally, we explore predictive models for directly estimating these traits from the text of the questions and answers. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningQuestion AnsweringSimilar Papers 제목 키워드 기반
Predicting Difficulty and Discrimination of Natural Language Questions
Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Down…
Active LearningQuestion AnsweringIntroducing a framework to assess newly created questions with Natural Language Processing
Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out pred…
Multiple-choiceWhich is Better for Deep Learning: Python or MATLAB? Answering Comparative Questions in Natural Language
We present a system for answering comparative questions (Is X better than Y with respect to Z?) in natural language. Answering such questions is important for assisting humans in making informed decisions. The key compon…
R2DE: a NLP approach to estimating IRT parameters of newly generated questions
The main objective of exams consists in performing an assessment of students' expertise on a specific subject. Such expertise, also referred to as skill or knowledge level, can then be leveraged in different ways (e.g., …
Multiple-choiceQuestion GenerationQuestion-GenerationMeasuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks
Accuracy-based evaluation of Large Language Models (LLMs) measures benchmark-specific performance rather than underlying medical competency: it treats all questions as equally informative, conflates model ability with it…