Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

2026-04-30

AI Summary

This study addresses the limitation of existing large language model (LLM) approaches to automated short-answer scoring, which rely on aggregate accuracy metrics and fail to capture performance disparities across student responses of varying difficulty. For the first time, item response theory (IRT) is introduced into LLM scoring evaluation, establishing a fine-grained analytical framework that models scoring correctness as a function of both rater ability and response difficulty. Integrating semantic embeddings and error pattern recognition, the authors evaluate 17 open-source LLMs on the SciEntsBank and Beetle datasets. Results reveal a significant decline in scoring accuracy as response difficulty increases. High-difficulty responses are frequently misclassified as “partially correct but incomplete” due to semantic divergence, strong contradictory signals, and embedding isolation, exposing underlying model fragility and intermediate-label collapse that aggregate metrics obscure.

Abstract

Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the \texttt{partially\_correct\_incomplete} label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures.

Problem

• automatic short answer grading
• large language models
• item response theory
• grading difficulty
• evaluation metrics

Innovation

• Item Response Theory
• Large Language Models
• Automatic Short Answer Grading
• Grading Difficulty
• Response-Level Evaluation