LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

2026-06-17

AI Summary

This study presents the first systematic evaluation of whether large language models (LLMs) can capture item discriminability—the core psychometric property that enables reading comprehension items to differentiate between students of varying ability levels. Under a zero-shot setting, 42 LLMs were assessed using two complementary approaches: direct prediction of item discriminability and classical test theory (CTT) calibration based on model-synthesized responses. Results indicate that although LLM outputs contain non-random signals, their discriminability patterns show limited alignment with human-calibrated benchmarks: the best Spearman correlation for direct prediction reaches only 0.152, improving modestly to 0.241 with the CTT approach—still far from reliably replicating the discrimination structure inherent in human responses. This work thus reveals significant limitations of current LLMs in psychometric modeling.

Abstract

Item discrimination is a fundamental psychometric property of educational assessment, which measures whether an item meaningfully distinguishes students with higher proficiency from students with lower proficiency. While various existing works have explored whether large language models (LLMs) can estimate item difficulty, it remains unclear whether they can capture item discrimination. In this work, we evaluate 42 proprietary and open-weight LLMs in zero-shot settings using two complementary approaches: direct discrimination prediction, where models explicitly estimate an item's discrimination value from its content, and response-based Classical Test Theory (CTT) calibration, where LLM answers are treated as synthetic student responses to compute discrimination scores. Our results show that direct prediction yields weak alignment with human-calibrated discrimination: the best-performing model reaches only a Spearman correlation of 0.152. Response-based CTT calibration provides a stronger but still limited signal, with the all-persona synthetic respondent pool reaching a Spearman correlation of 0.241. These findings highlight item discrimination as an open challenge for LLM-based psychometric evaluation: current LLMs contain non-random discrimination-relevant signal, but they do not yet reliably capture how assessment items distinguish human students.

Problem

• item discrimination
• large language models
• reading comprehension assessment
• psychometric evaluation
• educational assessment

Innovation

• item discrimination
• large language models
• psychometric evaluation
• Classical Test Theory
• educational assessment