LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model

2026-08-23

AI Summary

本文提出结合多维项目反应理论与问题上下文的方法，以预测大语言模型在未见问题上的表现，解决了仅用简单平均法评估可能混淆模型能力和题目特征的问题。

Abstract

Evaluation of large language models (LLMs) increasingly requires predicting how a model will perform on new questions or tasks before collecting large amounts of new annotations. This problem is challenging because question difficulty, scenario, and underlying capability demands can vary substantially. Simple retrospective averages may confound model ability with item characteristics. In this paper, we study a model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions. The framework represents LLMs through latent capability profiles while using question content to inform item characteristics, allowing information to transfer beyond previously observed items. Empirically, we find that for within-scenario evaluation, incorporating question embeddings improves prediction relative to model-free baselines, and that multidimensional latent structure provides a richer description of capability variation than unidimensional alternatives. At the same time, our results reveal an important limitation that the generalizability does not necessarily translate into reliable prediction under cross-scenario shift. These findings suggest that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation, while also highlighting cross-scenario generalization as a central open challenge.

Problem

• large language models
• unseen questions
• multidimensional item response theory
• question difficulty
• cross-scenario generalization

Innovation

• multidimensional IRT model
• contextual information
• latent capability profiles
• cross-scenario generalization