Do Deep Neural Nets Display Human-like Attention in Short Answer Scoring?
Deep Learning (DL) techniques have been increasingly adopted for Automatic Text Scoring in education. However, these techniques often suffer from their inabilities to explain and justify how a prediction is made, which, unavoidably, decreases their trustworthiness and hinders educators from embracing them in practice. This study aimed to investigate whether (and to what extent) DL-based graders align with human graders regarding the important words they identify when marking short answer questions. To this end, we first conducted a user study to ask human graders to manually annotate important words in assessing answer quality and then measured the overlap between these human-annotated words and those identified by DL-based graders (i.e., those receiving large attention weights). Furthermore, we ran a randomized controlled experiment to explore the impact of highlighting important words detected by DL-based graders on human grading. The results showed that: (i) DL-based graders, to a certain degree, displayed alignment with human graders no matter whether DL-based graders and human graders agreed on the quality of an answer; and (ii) it is possible to facilitate human grading by highlighting those DL-detected important words, though further investigations are necessary to understand how human graders exploit such highlighted words.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Product Title Refinement via Multi-Modal Generative Adversarial Learning
Nowadays, an increasing number of customers are in favor of using E-commerce Apps to browse and purchase products. Since merchants are usually inclined to employ redundant and over-informative product titles to attract c…
AttributeGenerative Adversarial Networkreinforcement-learningReinforcement Learning+1Assessing the nature of large language models: A caution against anthropocentrism
Generative AI models garnered a large amount of public attention and speculation with the release of OpenAIs chatbot, ChatGPT. At least two opinion camps exist: one excited about possibilities these models offer for fund…
ChatbotMulti-Modal Generative Adversarial Network for Short Product Title Generation in Mobile E-Commerce
Nowadays, more and more customers browse and purchase products in favor of using mobile E-Commerce Apps such as Taobao and Amazon. Since merchants are usually inclined to describe redundant and over-informative product t…
AttributeGenerative Adversarial NetworkReinforcement LearningThe progression of visual search in multiple item displays: First relational, then feature-based
It is well-known that visual attention can be tuned in a context-dependent manner to elementary features, such as searching for all redder items or the reddest item, supporting a relational theory of visual attention. Ho…
Realizing data features by deep nets
This paper considers the power of deep neural networks (deep nets for short) in realizing data features. Based on refined covering number estimates, we find that, to realize some complex data features, deep nets can impr…