Learning Item Embeddings and Hyperparameters for IRT Calibration via Monte Carlo EM

2026-07-07

AI Summary

This study addresses the challenge of inaccurate item parameter estimation for new items in computerized adaptive testing, which arises from sparse response data and degrades scoring quality. To tackle this issue, the authors propose a content-driven compact calibration method that leverages a pretrained shallow ReLU network to map handcrafted item features into a low-dimensional embedding space. This embedding is then decoupled and integrated with a linear, interpretable three-parameter logistic (3PL) item response theory (IRT) model, eliminating the need for online parameter tuning during deployment. The approach achieves, for the first time, a disentangled joint optimization of neural embeddings and interpretable IRT parameters. Evaluated on two task types in the Duolingo English Test, the method demonstrates superior performance using only a six-dimensional embedding, outperforming larger models and confirming its efficiency and effectiveness.

Abstract

High-stakes computerized adaptive tests (CATs) must continually calibrate new items in their item bank. When an item is new, few responses are available, so item parameter estimates -- and thus test scores -- are poor. Item features and explanatory item response theory (IRT) models mitigate this by folding item content into calibration. Neural IRT models, whose item parameters are neural-net outputs, are powerful, but tuning hyperparameters and architectures in real time while a CAT is scoring is impractical and threatens validity. We propose a pre-launch step that fits a neural net to produce low-dimensional item embeddings, so the production system can use a simple linear explanatory IRT model on top of them. We use a neural parameterization of the 3-parameter logistic (3PL) model in which a feature network maps each item's content features to a representation $h_j = z(x_j) \in \mathbb{R}^d$, from which the discrimination and difficulty $(a,b)$ follow generalized linear forms; the guessing parameter $c$ is fixed to a global constant to avoid identifiability issues. The feature network and latent abilities $θ$ are fit jointly via Monte Carlo Expectation-Maximization (MCEM), with no separate ability-estimation or pre-calibration stage. Using an item-split protocol that holds out entire items to simulate feature-only evaluation, we apply this to two Duolingo English Test practice task types -- yes/no vocabulary and vocabulary-in-context -- searching over feature sets, architectures, and dimensions $d$. A shallow two-layer ReLU network with $d=6$ and hand-engineered scalar features matches or beats larger architectures on held-out items for both. This is a first step toward a compact, content-derived item embedding for the Scalable Parametric Item Calibration Engine (SPICE), the fully Bayesian engine at the core of the S2A3 adaptive-testing system.

Problem

• computerized adaptive testing
• item calibration
• item response theory
• cold-start problem
• neural IRT

Innovation

• item embeddings
• neural IRT
• Monte Carlo EM
• explanatory item response theory
• computerized adaptive testing