Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

2026-06-14

AI Summary

This work addresses the high cost of pairwise human comparisons in subjective ranking tasks by proposing an efficient human-in-the-loop sorting framework. Instead of using vision-language models (VLMs) as annotator surrogates, the method employs a VLM as a priority scheduler guided by a novel composite surprise metric that integrates position-bias-corrected confidence, Elo rating gaps, and voting entropy. The framework directs merge sort to request human judgments only for highly uncertain comparisons, while inferring the rest via transitivity, and incorporates an adaptive budget allocation mechanism. Evaluated across six text and image benchmarks, the approach achieves a 6–12 point improvement in Kendall’s τ×100 over Active Elo under identical annotation budgets and skips up to 535 non-informative comparisons per round.

Abstract

Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons ($O(n^2)$). While sorting-based methods have reduced this burden to $O(n\log n)$, they still require expensive human judgment for every single comparison. To further improve annotation efficiency, we propose leveraging a Vision-Language Model (VLM) not as an annotator replacement, but as a \emph{question prioritizer} to identify which comparisons genuinely require human judgment. The proposed \textbf{Surprise-Guided MergeSort (SGS)} framework achieves this through three integrated components: (1) a bottom-up MergeSort scheduler that structures comparisons and exploits transitivity, (2) a composite Surprise Scorer -- combining position-bias-cancelled VLM confidence, Elo gap, and vote entropy -- to quantify comparison ambiguity, and (3) an adaptive budget allocator that routes high-surprise pairs to humans while automating low-surprise pairs via transitivity inference. Validation was conducted on six diverse benchmarks spanning text similarity (STS-B, BIOSSES, SICKR-STS) and image quality assessment (KonIQ-10k, TID2013, LIVE Challenge). SGS effectively identified and skipped up to 535 non-informative comparisons per session. Consequently, it achieved Kendall's $τ{\times}100$ improvements of $+6$ to $+12$ over Active Elo under the same total budget. These results demonstrate that combining VLM-guided surprise metrics with algorithmic sorting provides a generally consistent accuracy-efficiency trade-off across diverse domains.

Problem

• human-in-the-loop
• pairwise comparison
• annotation efficiency
• subjective ranking
• budget-efficient

Innovation

• Surprise-Guided MergeSort
• human-in-the-loop ranking
• Vision-Language Model
• adaptive comparison scheduling
• transitivity inference