🤖 AI Summary
This work addresses the susceptibility of conditional log-probability–based scoring in multiple-choice evaluation to answer length bias, which can unfairly penalize or overfavor longer responses. To mitigate this issue, the paper proposes Bayesian Accuracy—a plug-and-play scoring method grounded in Bayesian posterior probabilities. By explicitly modeling the prior distribution over answer lengths, the approach effectively eliminates linear length bias without requiring additional forward passes. Evaluated across diverse benchmarks and few-shot settings, Bayesian Accuracy substantially reduces empirical length bias and enhances the fairness and reliability of model assessment compared to standard accuracy and length-normalization baselines.
📝 Abstract
Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over-corrects, introducing a bias toward longer answers instead. We first analyze these scoring rules, characterizing when standard and length-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths. Motivated by this analysis, we introduce \emph{Bayesian accuracy}, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects. Bayesian accuracy is a drop-in replacement for likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.