The Initial Screening Order Problem

2023-07-28

arXiv.org

AI Summary

This study investigates the impact of Initial Screening Order (ISO) on fairness and selection quality in human candidate screening. We identify two distinct screening behaviors—best-k (optimizing for top candidates) and good-k (satisfying a quality threshold)—and propose the first computational model capturing time-varying inconsistency in human reviewers and its interaction with ISO. Our evaluation framework integrates position-bias analysis, quantification of both group-level and individual-level fairness, and behavioral simulation. Experimental results demonstrate that, under the good-k paradigm, ISO preserves group fairness but substantially degrades individual fairness and reduces top-k selection quality. This work provides the first systematic characterization of the latent bias mechanisms induced by ISO, establishing a theoretical foundation and a generalizable methodology for designing bias-mitigating screening strategies.

Abstract

We investigate the role of the initial screening order (ISO) in candidate screening. The ISO refers to the order in which the screener searches the candidate pool when selecting $k$ candidates. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its impact on the optimality and fairness of the selected $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener given an ISO: the best-$k$, where it selects the top $k$ candidates; and the good-$k$, where it selects the first good-enough $k$ candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time. Our analysis, in particular, shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for both screeners. Our simulation framework is flexible enough to account for multiple candidate screening tasks, being an alternative to running real-world procedures.

Problem

• Initial Screening Order
• Fairness
• Candidate Quality

Innovation

• Initial Screening Order
• Fairness in Selection
• Simulation Model for Human-like Screening