DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

2026-06-24

AI Summary

This work addresses the long-standing disconnect between static benchmarks and arena-style preference data in large language model evaluation, which has hindered a unified assessment of model capabilities. The authors propose a joint calibration framework based on latent variable modeling that maps both models and evaluation items into a shared latent space, simultaneously estimating model proficiency, item difficulty, and item discriminability. By integrating static correctness labels with open-ended preference signals, this approach achieves the first unified treatment of these two evaluation paradigms. Evaluated across 18 state-of-the-art models in four domains—coding, mathematics, domain-specific knowledge, and everyday queries—the framework yields robust performance rankings and enables downstream applications such as item-level interpretable diagnostics, efficient sampling, benchmark compression, and anomaly detection.

Abstract

Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.

Problem

• LLM evaluation
• static benchmarks
• arena-style preference
• model calibration
• unified evaluation

Innovation

• joint calibration
• unified evaluation
• latent space modeling
• benchmark compression
• preference learning