Predicting Difficulty and Discrimination of Natural Language Questions
Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Downing, 2003). More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets (Lalor et al., 2019; Vania et al., 2021; Rodriguez et al., 2021). In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningQuestion AnsweringSimilar Papers 제목 키워드 기반
Probing Difficulty and Discrimination of Natural Language Questions With Item Response Theory
Item Response Theory (IRT) has been extensively used to characterize question difficulty for human subjects in domains including cognitive psychology and education (Primi et al.,2014; Downing, 2003). In this work, we exp…
Active LearningQuestion AnsweringIntroducing a framework to assess newly created questions with Natural Language Processing
Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out pred…
Multiple-choiceThe Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory
High-quality test items are essential for educational assessments, particularly within Item Response Theory (IRT). Traditional validation methods rely on resource-intensive pilot testing to estimate item difficulty and d…
MathMultiple-choiceA Weak Supervision Approach for Predicting Difficulty of Technical Interview Questions
Predicting difficulty of questions is crucial for technical interviews. However, such questions are long-form and more open-ended than factoid and multiple choice questions explored so far for question difficulty predict…
Multiple-choicePredictionQuestion GenerationQuestion-GenerationUnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions
This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Ou…
Data AugmentationMultiple-choice