Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?
Estimating the difficulty of multiple-choice questions would be great help for educators who must spend substantial time creating and piloting stimuli for their tests, and for learners who want to practice. Supervised approaches to difficulty estimation have yielded to date mixed results. In this contribution we leverage an aspect of generative large models which might be seen as a weakness when answering questions, namely their uncertainty, and exploit it towards exploring correlations between two different metrics of uncertainty, and the actual student response distribution. While we observe some present but weak correlations, we also discover that the models' behaviour is different in the case of correct vs wrong answers, and that correlations differ substantially according to the different question types which are included in our fine-grained, previously unused dataset of 451 questions from a Biopsychology course. In discussing our findings, we also suggest potential avenues to further leverage model uncertainty as an additional proxy for item difficulty.
Code (1)
Tasks
Multiple-choiceSimilar Papers 제목 키워드 기반
Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension
Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question giv…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionAnswer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension
Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question giv…
Machine Reading ComprehensionMultiple-choiceReading ComprehensionA Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning
Proxy-based Deep Metric Learning (DML) learns deep representations by embedding images close to their class representatives (proxies), commonly with respect to the angle between them. However, this disregards the embeddi…
Metric LearningA Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies
Reliable uncertainty estimates are critical in safety-sensitive applications, where understanding the sources of predictive uncertainty is essential. This often requires disentangling epistemic uncertainty from aleatoric…
Out-of-Distribution DetectionEstimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
Large language models (LLMs) have progressed rapidly in complex reasoning and question answering, yet LLM hallucination remains a central bottleneck that hinders practical deployment, especially for commercial black-box …
Question Answering