Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions

2026-06-30

AI Summary

This study addresses the fundamental trade-off in large language model evaluation among evaluator coupling (γ), policy diversity (measured by entropy H), and few-shot reliability (quantified by the coefficient of variation CV). Extending empirical conditions from five to eleven, the work systematically quantifies the interplay among these three factors and introduces the first standardized benchmark dataset for evaluation. Results reveal a strong negative correlation between γ and H (r = −0.989), indicating that low coupling is accompanied by high measurement noise. Notably, no experimental setting simultaneously achieves γ < 0.2 and CV(N=5) < 0.3, highlighting an inherent tension among these desiderata. The analysis also uncovers anomalous patterns linked to version drift in GPT-4o, offering empirical grounding for the design of more robust and reliable evaluation frameworks.

Abstract

The bias-reliability tradeoff conjectures that LLM evaluation systems are constrained in (gamma, H, CV) space, where evaluator coupling (gamma), strategy diversity (H), and small-sample measurement reliability (CV(N)) cannot be simultaneously optimized at fixed sample size N. Prior evidence rests on n=5 conditions with complete metrics from a single study. We expand the empirical base to 11 conditions, measuring gamma and H for all 11 (nine with valid weight vectors) and CV(N=5) for seven with sufficient seeds (N >= 5). Five conditions provide the complete (gamma, H, CV) triple. The data confirm the trade-off: conditions with low evaluator coupling (gamma < 0.2) exhibit high measurement noise (CV(N=5) > 1.0), while conditions with strong coupling (gamma > 0.9) achieve low noise (CV(N=5) < 0.16). The correlation r(H, gamma) = -0.989 (n=5, excluding GPT-4o conditions) confirms that evaluator coupling suppresses strategy diversity. Four GPT-4o conditions show gamma=0.000 and H=1.000 across all seeds -- a pattern we attribute to version drift in the June 2026 GPT-4o API. No condition occupies the region {gamma < 0.2, CV(N=5) < 0.3}. We release all per-condition metrics as a standardized benchmark dataset for evaluator comparison.

Problem

• bias-reliability tradeoff
• evaluator coupling
• strategy diversity
• measurement reliability
• LLM evaluation

Innovation

• bias-reliability tradeoff
• evaluator coupling
• strategy diversity
• measurement reliability
• LLM evaluation