Can We Trust Item Response Theory for AI Evaluation?

2026-07-16

AI Summary

This study addresses the potential pitfalls of directly applying item response theory (IRT)—originally designed for human assessment—to the evaluation of artificial intelligence systems, where mismatched data-generating mechanisms may compromise inference validity. It presents the first systematic evaluation of IRT’s applicability to large language model benchmarks, examining the feasibility, scalability, and reliability of four estimation approaches—marginal maximum likelihood, Markov chain Monte Carlo (MCMC), variational inference, and neural pseudo-twin estimators—across 18,000 simulated conditions. The findings reveal that classical methods are computationally infeasible at scale, while scalable alternatives introduce bias when the number of models is small or their ability distribution deviates from normality. The work quantifies, for the first time, the failure boundaries of IRT in AI evaluation and establishes required sample sizes and diagnostic criteria for its reliable application.

Abstract

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate model capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regime of human testing, for which standard IRT estimation tools were originally developed: benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimation tools used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation conditions, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become infeasible in large benchmark settings, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or nonnormally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.

Problem

• Item Response Theory
• AI Evaluation
• Benchmarking
• Latent Trait Models
• Reliability

Innovation

• Item Response Theory
• AI Evaluation
• Large Language Models
• IRT Estimation
• Benchmarking Reliability