Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory

2026-04-21

AI Summary

This study addresses the lack of a clear performance ceiling for automated essay scoring (AES) systems on benchmarks contaminated by human scoring noise. Drawing on classical test theory, it proposes—without requiring additional annotations—a method to derive two dataset-specific upper bounds on Quadratic Weighted Kappa (QWK) from standard double-scored data: a theoretical upper bound, representing the maximum QWK achievable when a model perfectly predicts the true score, and a human-level upper bound, reflecting the QWK attainable when a model matches human inter-rater agreement. The work demonstrates that conventional human–human QWK often underestimates these true performance ceilings. Through both simulated and real-data experiments, the validity of the proposed bounds is confirmed, offering a principled benchmark for evaluating the current performance and potential of existing AES systems.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Automated essay scoring (AES) is commonly evaluated on public benchmarks using quadratic weighted kappa (QWK). However, because benchmark labels are assigned by human raters and inevitably contain scoring errors, it remains unclear both what QWK is theoretically attainable and what level is practically sufficient for deployment. We therefore derive two dataset-specific QWK ceilings based on the reliability concept in classical test theory, which can be estimated from standard two-rater benchmarks without additional annotation. The first is the theoretical ceiling: the maximum QWK that an ideal AES model that perfectly predicts latent true scores can achieve under label noise. The second is the human-like ceiling: the QWK attainable by an AES model with human-level scoring error, providing a practical target when AES is intended to replace a single human rater. We further show that human--human QWK, often used as a ceiling reference, can underestimate the true ceiling. Simulation experiments validate the proposed ceilings, and experiments on real benchmarks illustrate how they clarify the current performance and remaining headroom of modern AES models.

Problem

• Automated Essay Scoring
• Quadratic Weighted Kappa
• Classical Test Theory
• Scoring Reliability
• Human Rater Error

Innovation

• Automated Essay Scoring
• Quadratic Weighted Kappa
• Classical Test Theory
• Performance Ceiling
• Label Noise