WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays
This study presents WrAFT, a Writing Assessment and Feedback Tool, that delivers both accurate and reliable scores and effective comprehensive feedback to argumentative essays. WrAFT adopts a modular design by dividing automated writing evaluation (AWE) tasks into scoring, surface-level feedback, and deep-level feedback. In building the system, various Large Language Models (LLMs) have been evaluated, including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, through both direct prompting and supervised fine-tuning approaches. A proprietary dataset of 480 TOEFL Independent Writing essays with official benchmark scores was utilized. Benchmark-based evaluation shows that WrAFT achieves state-of-the-art performance in scoring, with a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 against official scores on a scale of 0-5. Human evaluation of system-generated feedback also reveals high approval ratings: 96.14 percent for surface-level feedback, 93.03 percent for deep-level macro feedback, and 94.69 percent for deep-level micro feedback. An interactive user interface has been developed for the system and is publicly available and free to use.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Neural Automated Writing Evaluation with Corrective Feedback
The utilization of technology in second language learning and teaching has become ubiquitous. For the assessment of writing specifically, automated writing evaluation (AWE) and grammatical error correction (GEC) have bec…
Automated Writing EvaluationGrammatical Error Correctionaw_nas: A Modularized and Extensible NAS framework
Neural Architecture Search (NAS) has received extensive attention due to its capability to discover neural network architectures in an automated manner. aw_nas is an open-source Python framework implementing various NAS …
Adversarial RobustnessNeural Architecture SearchThe NTNU-YZU System in the AESW Shared Task: Automated Evaluation of Scientific Writing Using a Convolutional Neural Network
Automated Evaluation of Writing -- 50 Years and Counting
In this theme paper, we focus on Automated Writing Evaluation (AWE), using Ellis Page{'}s seminal 1966 paper to frame the presentation. We discuss some of the current frontiers in the field and offer some thoughts on the…
Automated Writing EvaluationAutomated Evaluation for Student Argumentative Writing: A Survey
This paper surveys and organizes research works in an under-studied area, which we call automated evaluation for student argumentative writing. Unlike traditional automated writing evaluation that focuses on holistic ess…
Automated Writing EvaluationSurvey