Why Machines Misread Pedagogical Quality: Human-Machine Alignment in LLM-Based Pretest Question Evaluation

2026-06-22

AI Summary

This study addresses the challenges in designing large-scale, high-quality pretest items, where manual evaluation is time-consuming and struggles to balance openness, cognitive depth, and alignment with learning objectives, while automated scoring often exhibits systematic discrepancies with human judgment. The authors propose an AI-assisted workflow that integrates large language model–generated items, structured rubric-based assessment, and iterative refinement. Through a 2×2 controlled experiment, they investigate how rubric operationalization and evaluation modes affect human–AI agreement. Findings reveal systematic divergence between human and AI judgments; revising the rubric—rather than prompting models to justify responses first—more effectively improves consistency, and the two interventions are complementary. The work underscores the critical role of operationalizing instructional quality in enabling scalable AI-assisted pretest development and offers a reproducible methodological framework with empirical support for human–AI collaborative educational assessment.

Abstract

Designing effective pretest questions is challenging at scale: high-quality questions require careful calibration of openness, cognitive depth, and alignment with learning objectives, yet generating and evaluating them manually is time-consuming. We present an AI-assisted workflow for pretest question development that combines automated generation, rubric-based evaluation, and iterative selection. Because the workflow relies on machine evaluation to filter questions at scale, we investigate the alignment between human and machine judgments across a 2x2 design varying rubric operationalization and evaluation mode. Our findings show that human-machine disagreements are systematic rather than random, that rubric revision has a larger effect on alignment than rationale-first evaluation, and that the two interventions are complementary. These findings highlight that scalable AI-assisted pretesting depends not only on generation capability but on how pedagogical quality is operationalized for machine interpretation.

Problem

• pretest question evaluation
• human-machine alignment
• pedagogical quality
• LLM-based assessment
• rubric operationalization

Innovation

• human-machine alignment
• rubric operationalization
• LLM-based evaluation
• pretest question generation
• pedagogical quality assessment