A Principled Approach to Randomized Selection under Uncertainty

2025-06-23

AI Summary

To address the limited robustness of deterministic selection mechanisms under evaluation errors and outcome uncertainty, this paper proposes MERIT, a randomized selection framework based on quality interval estimation. MERIT optimizes for the maximum expected number of high-quality candidates selected in the worst-case scenario. It is the first method to simultaneously satisfy key axiomatic properties—interpretability, monotonicity, and robustness. By modeling uncertainty via quality confidence intervals, MERIT formulates an efficiently solvable randomized selection problem and provides a polynomial-time algorithm scalable to candidate sets exceeding 10,000. Experiments demonstrate that MERIT significantly outperforms existing approaches in worst-case performance while maintaining comparable expected utility. Thus, it delivers a principled, scalable, and theoretically guaranteed robust solution for high-stakes decision-making scenarios such as scientific peer review and talent selection.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Many decision-making processes involve evaluating and then selecting items; examples include scientific peer review, job hiring, school admissions, and investment decisions. The eventual selection is performed by applying rules or deliberations to the raw evaluations, and then deterministically selecting the items deemed to be the best. These domains feature error-prone evaluations and uncertainty about future outcomes, which undermine the reliability of such deterministic selection rules. As a result, selection mechanisms involving explicit randomization that incorporate the uncertainty are gaining traction in practice. However, current randomization approaches are ad hoc, and as we prove, inappropriate for their purported objectives. In this paper, we propose a principled framework for randomized decision-making based on interval estimates of the quality of each item. We introduce MERIT (Maximin Efficient Randomized Interval Top-k), an optimization-based method that maximizes the worst-case expected number of top candidates selected, under uncertainty represented by overlapping intervals (e.g., confidence intervals or min-max intervals). MERIT provides an optimal resource allocation scheme under an interpretable notion of robustness. We develop a polynomial-time algorithm to solve the optimization problem and demonstrate empirically that the method scales to over 10,000 items. We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches. Finally, we empirically compare algorithms on synthetic peer review data. Our experiments demonstrate that MERIT matches the performance of existing algorithms in expected utility under fully probabilistic review data models used in previous work, while outperforming previous methods with respect to our novel worst-case formulation.

Problem

• Addresses unreliable deterministic selection under evaluation uncertainty
• Proposes MERIT for optimal randomized selection using interval estimates
• Ensures robustness and efficiency in top-k candidate selection

Innovation

• Interval estimates for randomized decision-making
• MERIT optimizes worst-case top candidate selection
• Polynomial-time algorithm scales to 10,000 items