WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays

2026-07-15

AI Summary

This work proposes a modular Automated Writing Evaluation (AWE) system designed to deliver high-accuracy scoring and multi-level feedback for TOEFL Independent Writing tasks. The system decomposes the task into three integrated modules—scoring, surface-level feedback, and deep-level feedback—marking the first publicly available framework to unify holistic scoring with fine-grained feedback generation. Trained on 480 officially scored essays using large language models including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, the approach leverages both direct prompting and supervised fine-tuning strategies. Experimental results demonstrate strong performance, achieving a Quadratic Weighted Kappa (QWK) of 0.84 (RMSE = 0.44) for scoring accuracy and high human-rated satisfaction rates: 96.14% for surface feedback, 93.03% for macro-level deep feedback, and 94.69% for micro-level deep feedback. The system is now publicly accessible and freely available.

Abstract

This study presents WrAFT, a Writing Assessment and Feedback Tool, that delivers both accurate and reliable scores and effective comprehensive feedback to argumentative essays. WrAFT adopts a modular design by dividing automated writing evaluation (AWE) tasks into scoring, surface-level feedback, and deep-level feedback. In building the system, various Large Language Models (LLMs) have been evaluated, including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, through both direct prompting and supervised fine-tuning approaches. A proprietary dataset of 480 TOEFL Independent Writing essays with official benchmark scores was utilized. Benchmark-based evaluation shows that WrAFT achieves state-of-the-art performance in scoring, with a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 against official scores on a scale of 0-5. Human evaluation of system-generated feedback also reveals high approval ratings: 96.14 percent for surface-level feedback, 93.03 percent for deep-level macro feedback, and 94.69 percent for deep-level micro feedback. An interactive user interface has been developed for the system and is publicly available and free to use.

Problem

• Automated Writing Evaluation
• Argumentative Essays
• Writing Assessment
• Feedback Generation
• Essay Scoring

Innovation

• modularized AWE
• large language models
• argumentative essay evaluation
• supervised fine-tuning
• comprehensive feedback