Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring

2026-07-13

AI Summary

This study addresses the limitation of traditional aggregative reliability estimates, which overlook heterogeneity in measurement burden across response conditions and may thus misjudge measurement adequacy for subpopulations. The authors propose a conditional generalizability framework that, for the first time, incorporates entropy as an operational stratification variable within generalizability theory. By integrating automated scoring configuration space scanning, analytical D-studies, and response entropy stratification, the framework establishes a method for assessing condition-dependent reliability tailored to AI-based scoring systems. Empirical results reveal high overall dependency (Phi ≈ 0.76), with robust—though slightly reduced—dependency persisting in high-entropy response strata (Phi = 0.84–0.88). These findings demonstrate the non-uniform distribution of dependency across response conditions, offering empirical grounding for differentiated decision-making in assessment contexts.

Abstract

Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata. This study introduces a conditional generalizability framework with three components. First, automated scoring configurations -- the encoder architectures and scoring-head families admissible within a fixed pipeline -- are treated as a universe of admissible measurement conditions rather than incidental modeling choices. Second, analytical D-study projections are compared with empirical configuration sweeps over a finite scoring pool, yielding two estimands of design adequacy whose agreement or divergence diagnoses the realized configuration universe. Third, evidence is conditioned on entropy-defined response strata, treating entropy as an operational stratification variable, not a construct claim about writing quality. Whereas recent generalizability-theory extensions address AI-generated item variants on the response side, this framework addresses the analogous scoring-side problem: AI-mediated scoring configurations. Demonstrated with automated essay scoring of timed L2 writing, the realized design was dependable in aggregate (Phi approx 0.76). Re-estimated within entropy strata, dependability stayed high but declined modestly and robustly (Phi = 0.88, 0.87, 0.84) -- a gradient implying different decision-study requirements, the highest-entropy stratum requiring the most crossed conditions. The framework offers a portable workflow for evaluating nonuniform dependability.

Problem

• nonuniform dependability
• response conditions
• automated essay scoring
• generalizability theory
• measurement heterogeneity

Innovation

• conditional generalizability
• automated essay scoring
• measurement dependability
• entropy-based stratification
• scoring configuration