An Interpretable and Scalable Framework for Evaluating Large Language Models

2026-05-07

AI Summary

This work addresses critical limitations in current large language model (LLM) evaluation methods, which rely on average accuracy while neglecting output stochasticity and item heterogeneity, and suffer from the computational expense and numerical instability of traditional approaches based on Item Response Theory (IRT). To overcome these challenges, the authors propose the first evaluation framework that jointly ensures interpretability and scalability. By leveraging the majorization–minimization principle, the method reformulates the joint modeling of model ability and item characteristics as a constrained matrix factorization subproblem, enabling efficient and stable estimation. Evaluated on MATH-500 and six Open LLM Leaderboard benchmarks, the approach achieves speedups of several orders of magnitude over existing methods while maintaining comparable or superior accuracy, and further uncovers intrinsic patterns in item difficulty and discrimination.

Abstract

Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and offer insights into item difficulty and discrimination, informing more principled benchmark design.

Problem

• Large Language Models
• Evaluation
• Item Response Theory
• Scalability
• Interpretability

Innovation

• Item Response Theory
• majorization-minimization
• matrix factorization
• scalable evaluation
• interpretable AI