Misspecifying non-compensatory as compensatory IRT: analysis of estimated skills and variance

2025-07-20

AI Summary

This study investigates the bias in ability estimation and inflation of parameter uncertainty arising from misspecifying a noncompensatory multidimensional item response theory (MIRT) model as compensatory. Using asymptotic variance analysis and theoretical derivation, we establish— for the first time—that ability estimates are systematically overestimated near the origin, and characterize the bias as bimodal: overestimation for low-ability examinees and underestimation for high-ability examinees. We derive the distributional properties of the estimation bias under misspecification and obtain a theoretical upper bound on the asymptotic variance of ability estimates. Results demonstrate that model misspecification not only distorts point estimates but also substantially amplifies estimation uncertainty—particularly when latent dimensions exhibit strong interaction. Our findings provide quantifiable theoretical criteria and diagnostic tools for MIRT model selection, thereby enhancing the robustness and scientific rigor of multidimensional test modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Graphical Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Multidimensional item response theory is a statistical test theory used to estimate the latent skills of learners and the difficulty levels of problems based on test results. Both compensatory and non-compensatory models have been proposed in the literature. Previous studies have revealed the substantial underestimation of higher skills when the non-compensatory model is misspecified as the compensatory model. However, the underlying mechanism behind this phenomenon has not been fully elucidated. It remains unclear whether overestimation also occurs and whether issues arise regarding the variance of the estimated parameters. In this paper, we aim to provide a comprehensive understanding of both underestimation and overestimation through a theoretical approach. In addition to the previously identified underestimation of the skills, we newly discover that the overestimation of skills occurs around the origin. Furthermore, we investigate the extent to which the asymptotic variance of the estimated parameters differs when considering model misspecification compared to when it is not taken into account.

Problem

• Investigates skill underestimation in misspecified non-compensatory IRT models
• Examines overestimation of skills near the origin in misspecified models
• Analyzes variance differences in estimated parameters due to misspecification

Innovation

• Theoretical analysis of skill underestimation and overestimation
• Investigates variance differences under model misspecification
• Examines non-compensatory IRT misspecified as compensatory