ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities

2025-08-19

AI Summary

Large language models (LLMs) lack systematic evaluation of their ability to align student cognitive development stages (SCA) with text difficulty in Chinese educational contexts—particularly under Vygotsky’s Zone of Proximal Development (ZPD) theory. Method: We introduce ZPD-SCA, the first ZPD-grounded, stage-wise Chinese reading comprehension difficulty benchmark, constructed from expert teacher annotations and evaluated via zero-shot and in-context learning paradigms across mainstream LLMs. Contribution/Results: Experiments reveal that most LLMs perform below random chance in zero-shot settings; in-context learning nearly doubles accuracy but exposes pronounced directional bias and cross-genre instability. This work is the first to empirically uncover systematic misalignment and emergent alignment capabilities of LLMs in pedagogical judgment, establishing a novel benchmark and methodology for personalized learning resource recommendation and educational LLM evaluation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have demonstrated potential in educational applications, yet their capacity to accurately assess the cognitive alignment of reading materials with students' developmental stages remains insufficiently explored. This gap is particularly critical given the foundational educational principle of the Zone of Proximal Development (ZPD), which emphasizes the need to match learning resources with Students' Cognitive Abilities (SCA). Despite the importance of this alignment, there is a notable absence of comprehensive studies investigating LLMs' ability to evaluate reading comprehension difficulty across different student age groups, especially in the context of Chinese language education. To fill this gap, we introduce ZPD-SCA, a novel benchmark specifically designed to assess stage-level Chinese reading comprehension difficulty. The benchmark is annotated by 60 Special Grade teachers, a group that represents the top 0.15% of all in-service teachers nationwide. Experimental results reveal that LLMs perform poorly in zero-shot learning scenarios, with Qwen-max and GLM even falling below the probability of random guessing. When provided with in-context examples, LLMs performance improves substantially, with some models achieving nearly double the accuracy of their zero-shot baselines. These results reveal that LLMs possess emerging abilities to assess reading difficulty, while also exposing limitations in their current training for educationally aligned judgment. Notably, even the best-performing models display systematic directional biases, suggesting difficulties in accurately aligning material difficulty with SCA. Furthermore, significant variations in model performance across different genres underscore the complexity of task. We envision that ZPD-SCA can provide a foundation for evaluating and improving LLMs in cognitively aligned educational applications.

Problem

• Assessing cognitive alignment of reading materials with student development
• Evaluating reading comprehension difficulty across different age groups
• Identifying limitations in LLMs' educational judgment capabilities

Innovation

• Introduces ZPD-SCA benchmark for reading difficulty assessment
• Uses expert teacher annotations for cognitive alignment evaluation
• Tests LLMs with zero-shot and in-context learning scenarios