Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring
Automated essay scoring (AES) aims to score essays written for a given prompt, which defines the writing topic. Most existing AES systems assume to grade essays of the same prompt as used in training and assign only a holistic score. However, such settings conflict with real-education situations; pre-graded essays for a particular prompt are lacking, and detailed trait scores of sub-rubrics are required. Thus, predicting various trait scores of unseen-prompt essays (called cross-prompt essay trait scoring) is a remaining challenge of AES. In this paper, we propose a robust model: prompt- and trait relation-aware cross-prompt essay trait scorer. We encode prompt-aware essay representation by essay-prompt attention and utilizing the topic-coherence feature extracted by the topic-modeling mechanism without access to labeled data; therefore, our model considers the prompt adherence of an essay, even in a cross-prompt setting. To facilitate multi-trait scoring, we design trait-similarity loss that encapsulates the correlations of traits. Experiments prove the efficacy of our model, showing state-of-the-art results for all prompts and traits. Significant improvements in low-resource-prompt and inferior traits further indicate our model's strength.
Code (1)
Tasks
Automated Essay ScoringRelationSimilar Papers 제목 키워드 기반
Scaling Personality Control in LLMs with Big Five Scaler Prompts
We present Big5-Scaler, a prompt-based framework for conditioning large language models (LLMs) with controllable Big Five personality traits. By embedding numeric trait values into natural language prompts, our method en…
Dialogue GenerationTrait-Aware Policy Optimization for Autoregressive Multi-Trait Essay Scoring
Multi-trait essay scoring aims to provide fine-grained evaluation of writing quality across multiple dimensions. However, how to effectively post-train autoregressive scoring models remains underexplored. In this paper, …
TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring
Research on holistic Automated Essay Scoring (AES) is long-dated; yet, there is a notable lack of attention for assessing essays according to individual traits. In this work, we propose TRATES, a novel trait-specific and…
Automated Essay ScoringLanguage ModelingLanguage ModellingLarge Language ModelPsyAgent: Constructing Human-like Agents Based on Psychological Modeling and Contextual Interaction
Human-like agents must express stable dispositions while adapting to roles, relationships, and norms. We present PsyAgent, a schema-first framework that operationalizes the trait-context interface by coupling a Big Five …
Prompt EngineeringEvaluating LLM Alignment on Personality Inference from Real-World Interview Data
Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding, such as emotional support agents, counselors, and decision-making assistants. However, their ability to inter…