🤖 AI Summary
Current evaluations of large language models as judges (LLM-as-a-Judge) predominantly rely on single scalar metrics, overlooking their systematic biases and psychometric properties. This work proposes the Judge Datasheet protocol, treating LLM judges as measurement instruments and systematically characterizing their behavior through controlled experiments—including vacuum inputs, positional perturbations, and graded quality responses—augmented by novel concepts such as “dark current” and “direction–stability decomposition.” Evaluations across three open-source models reveal significant differences: Llama-3.1-8B exhibits high dark current and conflicting positional preferences, whereas Qwen2.5-32B demonstrates superior performance. Furthermore, prompt engineering is found to modulate only the decision threshold, not the underlying discriminative capability.
📝 Abstract
LLM-as-a-judge systems are now routinely used for open-ended model evaluation, where human preference annotation is costly, slow, and difficult to reproduce. Yet these judges are often reported as scalar accuracy, win-rate, or agreement devices. We argue that a judge should instead be reported as a measurement instrument. We introduce a Judge Datasheet protocol that measures dark current under true-vacuum inputs, stable cross-sensitivity to same-quality surface variation, positional false preference, target sensitivity on a controlled quality ladder, and the criterion or operating point induced by tie instructions. The direction-stability decomposition reveals that apparent Delta0 preference can be stable surface response or disguised position bias. In a three-judge open-weight case study, Llama-3.1-8B shows high dark current and presentation-conflicted Delta0 behavior, Qwen2.5-14B is vacuum-clean and target-sensitive but mixes stable and positional over-discrimination, and Qwen2.5-32B is vacuum-clean with low stable cross-sensitivity and low positional false preference. A strict tie criterion eliminates Qwen32B Delta0 false preference but absorbs marginal Delta1 target signals into ties while preserving Delta5 sensitivity. The results show that prompting moves the criterion, not the resolution. We do not claim that the downstream mechanism hypothesis that motivated this work is confirmed; the contribution is a metrological protocol for measuring the measuring device before downstream claims are made.