LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models

2025-05-13

AI Summary

Existing zero-shot automated essay scoring (AES) methods rely on large language models (LLMs) to directly predict absolute scores, rendering them susceptible to model biases and inconsistent scoring behavior—leading to substantial discrepancies with human annotations. To address this, we propose an LLM-driven pairwise comparison framework for zero-shot AES: the scoring task is reformulated as relative preference judgment between essay pairs; LLMs perform preference inference to generate pairwise comparisons; and RankNet learns a total-order ranking to derive calibrated scores. This work is the first to formulate zero-shot AES as an LLM-based pairwise comparison problem, departing fundamentally from conventional absolute-scoring paradigms. Our approach achieves significant improvements over state-of-the-art zero-shot methods across multiple benchmarks, with marked gains in scoring accuracy, high computational efficiency, compatibility with diverse LLM backbones, and strong robustness and practicality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advances in large language models (LLMs) have enabled zero-shot automated essay scoring (AES), providing a promising way to reduce the cost and effort of essay scoring in comparison with manual grading. However, most existing zero-shot approaches rely on LLMs to directly generate absolute scores, which often diverge from human evaluations owing to model biases and inconsistent scoring. To address these limitations, we propose LLM-based Comparative Essay Scoring (LCES), a method that formulates AES as a pairwise comparison task. Specifically, we instruct LLMs to judge which of two essays is better, collect many such comparisons, and convert them into continuous scores. Considering that the number of possible comparisons grows quadratically with the number of essays, we improve scalability by employing RankNet to efficiently transform LLM preferences into scalar scores. Experiments using AES benchmark datasets show that LCES outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. Moreover, LCES is robust across different LLM backbones, highlighting its applicability to real-world zero-shot AES.

Problem

• Zero-shot automated essay scoring via pairwise comparisons
• Addressing model biases in direct absolute score generation
• Improving scalability with RankNet for large essay sets

Innovation

• Uses pairwise comparisons for essay scoring
• Employs RankNet for scalable score conversion
• Robust across different LLM backbones