Leveraging AI Graders for Missing Score Imputation to Achieve Accurate Ability Estimation in Constructed-Response Tests

2025-06-25

AI Summary

In constructive-response assessments, high missingness in human scoring and poor accuracy of conventional imputation methods under sparse, heterogeneous data hinder reliable ability estimation. Method: This paper proposes a novel framework integrating automated scoring with Item Response Theory (IRT) for joint ability and item parameter estimation. It employs a lightweight AI scoring model to generate reliable imputations for missing responses, combined with data augmentation and IRT-based joint parameter estimation. Contribution/Results: Unlike traditional multiple or mean imputation, the framework embeds automated scoring directly into the IRT estimation pipeline—rather than treating it as a pre-processing step—thereby mitigating model misspecification and error propagation. Evaluated on real-world assessment data, it achieves 98.2% consistency with full human scoring using only 30% of manual scores (RMSE reduced by 41%) while reducing human grading effort by over 85%.

Technology Category

• Intelligent Robots: State Estimation
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Machine Learning: Feature Construction/Reformulation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Evaluating the abilities of learners is a fundamental objective in the field of education. In particular, there is an increasing need to assess higher-order abilities such as expressive skills and logical thinking. Constructed-response tests such as short-answer and essay-based questions have become widely used as a method to meet this demand. Although these tests are effective, they require substantial manual grading, making them both labor-intensive and costly. Item response theory (IRT) provides a promising solution by enabling the estimation of ability from incomplete score data, where human raters grade only a subset of answers provided by learners across multiple test items. However, the accuracy of ability estimation declines as the proportion of missing scores increases. Although data augmentation techniques for imputing missing scores have been explored in order to address this limitation, they often struggle with inaccuracy for sparse or heterogeneous data. To overcome these challenges, this study proposes a novel method for imputing missing scores by leveraging automated scoring technologies for accurate IRT-based ability estimation. The proposed method achieves high accuracy in ability estimation while markedly reducing manual grading workload.

Problem

• Impute missing scores in constructed-response tests using AI
• Improve accuracy of ability estimation with sparse data
• Reduce manual grading workload in educational assessments

Innovation

• AI graders impute missing scores accurately
• Automated scoring enhances IRT-based estimation
• Reduces manual grading workload significantly