Statistical and computational challenges in ranking

2025-12-24

AI Summary

This paper addresses the problem of statistically optimal and computationally efficient ranking of expert abilities in general isotonic crowdsourcing models, based on their answer accuracy. It formalizes ranking as estimating an unknown row permutation π* that orders the rows of the crowdsourcing matrix M monotonically by underlying ability. We establish, for the first time, that no statistical–computational gap exists for this task—refuting the conventional belief that statistical optimality and polynomial-time computability are inherently separated under standard assumptions. Our method integrates low-degree polynomial testing, submatrix detection and estimation theory, isotropic structural analysis, and permutation estimation algorithms to construct a polynomial-time estimator achieving statistical minimax optimality. We further derive tight computational lower bounds and characterize sharp phase-transition thresholds for exact recovery. Collectively, this work establishes a theoretically complete and algorithmically feasible paradigm for crowdsourcing-based ranking.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Stochastic Optimization
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We consider the problem of ranking $n$ experts according to their abilities, based on the correctness of their answers to $d$ questions. This is modeled by the so-called crowd-sourcing model, where the answer of expert $i$ on question $k$ is modeled by a random entry, parametrized by $M_{i,k}$ which is increasing linearly with the expected quality of the answer. To enable the unambiguous ranking of the experts by ability, several assumptions on $M$ are available in the literature. We consider here the general isotonic crowd-sourcing model, where $M$ is assumed to be isotonic up to an unknown permutation $π^*$ of the experts - namely, $M_{π^{*-1}(i),k} geq M_{π^{*-1}(i+1),k}$ for any $iin [n-1], k in [d]$. Then, ranking experts amounts to constructing an estimator of $π^*$. In particular, we investigate here the existence of statistically optimal and computationally efficient procedures and we describe recent results that disprove the existence of computational-statistical gaps for this problem. To provide insights on the key ideas, we start by discussing simpler and yet related sub-problems, namely sub-matrix detection and estimation. This corresponds to specific instances of the ranking problem where the matrix $M$ is constrained to be of the form $λmathbf 1{S imes T}$ where $Ssubset [n], Tsubset [d]$. This model has been extensively studied. We provide an overview of the results and proof techniques for this problem with a particular emphasis on the computational lower bounds based on low-degree polynomial methods. Then, we build upon this instrumental sub-problem to discuss existing results and algorithmic ideas for the general ranking problem.

Problem

• Ranking experts by ability using their question-answer correctness data
• Developing optimal procedures under isotonic crowd-sourcing model assumptions
• Investigating computational-statistical trade-offs for ranking estimation problems

Innovation

• General isotonic crowd-sourcing model for ranking experts
• Computational lower bounds via low-degree polynomial methods
• Disproving computational-statistical gaps in ranking procedures