From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring

2026-03-06

AI Summary

This study addresses the lack of systematic comparison among large language model (LLM)-based automated essay scoring (AES) approaches tailored for English as a second language (L2) writing, particularly regarding their trade-offs in accuracy, cost, and robustness. The authors establish the first unified benchmark for IELTS Writing Task 2 and comprehensively evaluate four prominent LLM-based AES paradigms: encoder fine-tuning, prompt engineering, instruction tuning combined with retrieval-augmented generation (RAG), and supervised fine-tuning integrated with direct preference optimization (DPO) and RAG. Their experiments reveal fundamental trade-offs among accuracy, efficiency, and stability across these methods. The optimal configuration—k-shot supervised fine-tuning with RAG—achieves an F1 score of 93%, significantly outperforming baseline approaches and demonstrating the effectiveness and superiority of integrating fine-tuning with retrieval-augmented strategies for L2 AES.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have recently reshaped Automated Essay Scoring (AES), yet prior studies typically examine individual techniques in isolation, limiting understanding of their relative merits for English as a Second Language (L2) writing. To bridge this gap, we presents a comprehensive comparison of major LLM-based AES paradigms on IELTS Writing Task~2. On this unified benchmark, we evaluate four approaches: (i) encoder-based classification fine-tuning, (ii) zero- and few-shot prompting, (iii) instruction tuning and Retrieval-Augmented Generation (RAG), and (iv) Supervised Fine-Tuning combined with Direct Preference Optimization (DPO) and RAG. Our results reveal clear accuracy-cost-robustness trade-offs across methods, the best configuration, integrating k-SFT and RAG, achieves the strongest overall results with F1-Score 93%. This study offers the first unified empirical comparison of modern LLM-based AES strategies for English L2, promising potential in auto-grading writing tasks. Code is public at https://github.com/MinhNguyenDS/LLM_AES-EnL2

Problem

• Automated Essay Scoring
• Large Language Models
• Second Language Writing
• Comparative Study
• IELTS Writing

Innovation

• Automated Essay Scoring
• Large Language Models
• Preference Optimization
• Retrieval-Augmented Generation
• Supervised Fine-Tuning