Optimal Selection Using Algorithmic Rankings with Side Information

2025-11-06

AI Summary

This paper studies the optimal sequential selection of hidden-quality candidates by agents in online labor markets, where agents observe noisy ordinal rankings and binary availability signals (“available/busy”). Although busyness positively correlates with candidate quality, it reduces matching feasibility—introducing a trade-off between value and availability. Using game-theoretic modeling, Bayesian decision analysis, and signaling mechanism design, the paper characterizes agents’ equilibrium sequential selection strategies. It identifies a counterintuitive result: improving ranking accuracy can *reduce* social welfare—by amplifying agents’ tendency to misselect high-quality but busy candidates and increasing the likelihood that lower-ranked yet available candidates are overlooked. This challenges the conventional wisdom that “more accurate rankings are always better,” and provides theoretical foundations for designing platform signaling schemes and optimizing matching mechanisms.

Technology Category

• Game Theory and Economic Paradigms: Mechanism Design
• Machine Learning: Learning Preferences or Rankings
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Motivated by online platforms such as job markets, we study an agent choosing from a list of candidates, each with a hidden quality that determines match value. The agent observes only a noisy ranking of the candidates plus a binary signal that indicates whether each candidate is"free"or"busy."Being busy is positively correlated with higher quality, but can also reduce value due to decreased availability. We study the agent's optimal selection problem in the presence of ranking noise and free-busy signals and ask how the accuracy of the ranking tool impacts outcomes. In a setting with one high-valued candidate and an arbitrary number of low-valued candidates, we show that increased accuracy of the ranking tool can result in reduced social welfare. This can occur for two reasons: agents may be more likely to make offers to busy candidates, and (paradoxically) may be more likely to select lower-ranked candidates when rankings are more indicative of quality. We further discuss conditions under which these results extend to more general settings.

Problem

• Optimizing candidate selection using noisy rankings and binary availability signals
• Analyzing how ranking accuracy affects social welfare in selection systems
• Investigating paradoxical welfare reduction when rankings better indicate quality

Innovation

• Algorithmic rankings with binary side information
• Analysis of ranking accuracy impact on welfare
• Modeling selection trade-offs between quality signals