Problem
Research questions and friction points this paper is trying to address.
multiple-choice evaluation
order sensitivity
label-free strategies
positional bias
model accuracy
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
🤖 AI Summary
This study addresses the susceptibility of model performance in multiple-choice question evaluation to answer option ordering, which often obscures true knowledge assessment. The authors systematically investigate a “label-free answering” strategy—evaluating responses by either generation-matching or independent scoring after removing option labels—to mitigate positional bias. They innovatively decouple label hiding from the matching mechanism and conduct experiments combining cyclic permutations with large language model (LLM) matchers. Results reveal that baseline performance is recoverable only when models observe all options and are assisted by an LLM matcher. Contrary to common intuition, completely eliminating positional information does not consistently improve accuracy; instead, cyclic permutation frequently yields performance gains, challenging the assumption that position-invariance inherently leads to better evaluation fidelity.
📝 Abstract
Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.
Innovation
Methods, ideas, or system contributions that make the work stand out.
label-free evaluation
order sensitivity
positional bias
multiple-choice QA
LLM debiasing
🔎 Similar Papers
No similar papers found.
