Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking

2026-02-04

AI Summary

This work proposes a large language model (LLM)-based approach to optimize reviewer allocation in major machine learning conferences, where current practices often rely on random assignment or affinity-based heuristics that struggle to accurately identify papers near the acceptance threshold. By leveraging LLMs to perform pairwise comparisons of submissions and integrating these judgments into a Bradley–Terry model, the method constructs a ranking that predicts the borderline paper band without requiring human reviews. The authors further introduce an “expected impact” metric that jointly considers the overlap (ρ) between predicted and true borderline regions and the marginal decision value (Δ) gained from additional reviews. This metric guides the targeted allocation of limited extra reviewing resources to papers most likely to have their acceptance outcomes altered by additional scrutiny. Retrospective experiments demonstrate the effectiveness of the proposed framework.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper argues that large ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics. We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band \emph{before} human reviewing and to allocate \emph{marginal} reviewer capacity at assignment time. Concretely, given a venue-specific minimum review target (e.g., 3 or 4), we use this signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for accept/reject. We provide a simple expected-impact calculation in terms of (i) the overlap between the predicted and true borderline sets ($\rho$) and (ii) the incremental value of an extra review near the boundary ($\Delta$), and we provide retrospective proxies to estimate these quantities.

Problem

• marginal review allocation
• borderline papers
• acceptance boundary
• reviewer capacity
• LLM comparative ranking

Innovation

• LLM comparative ranking
• borderline paper identification
• review allocation
• Bradley-Terry model
• marginal review capacity