LLM Performance on a Real, Double-Marked GCSE Benchmark

2026-06-23

AI Summary

This study evaluates whether large language models (LLMs) can achieve or even exceed the inter-rater reliability of human examiners in authentic GCSE marking tasks. We construct a large-scale dataset comprising 32,534 student mock examination scripts, each doubly marked across five subjects and including handwritten responses, thereby enabling the first systematic assessment of off-the-shelf LLMs on both subjective questions (e.g., English essays) and complex handwritten mathematical solutions within a real-world, doubly marked educational evaluation setting. Results demonstrate that state-of-the-art LLMs exhibit high agreement with human markers across all subjects, significantly surpassing human inter-rater consistency. Notably, this performance shows minimal dependence on model scale, highlighting the strong cost-effectiveness and practical applicability of LLMs for automated scoring in educational assessment.

Abstract

We introduce a dataset of 32,534 double-marked real student responses to GCSE mock exams (GCSEs are the UK's national exams, taken at age ~16), spanning 328 questions across five subjects and including handwritten work. We test whether off-the-shelf large language models agree with examiners as closely as the two examiners agree with each other. We find that models overwhelmingly agree well with the examiner consensus across subjects, with the top performing models agreeing more closely with examiners than examiners agree with each other. Models achieve high scores for subjective tasks like English essay marking, as well as handling complex and messy handwritten Maths paper scripts. Agreement is uniform near the examiner line, and not massively discriminated by model size, providing cost-effective automated marking solutions.

Problem

• large language models
• automated marking
• GCSE exams
• examiner agreement
• handwritten responses

Innovation

• large language models
• automated marking
• double-marked dataset
• GCSE benchmark
• handwritten script evaluation