hyperFA*IR: A hypergeometric approach to fair rankings with finite candidate pool

2025-06-17

AI Summary

To address inaccurate fairness evaluation for small protected groups in top-k rankings under limited candidate pools, this work introduces the hypergeometric distribution into fair ranking modeling—rectifying bias inherent in conventional binomial assumptions that ignore sampling-without-replacement. We propose a parameter-free Monte Carlo statistical test for high-precision fairness assessment and design a weighted sampling-without-replacement equalization strategy that satisfies fairness constraints while preserving ranking utility. Experiments on multiple real-world datasets demonstrate that our method reduces misclassification rates for small-group fairness by 37%–62% over baselines, achieving statistically significant improvements. Moreover, the framework supports interpretable and intervention-aware fairness optimization, enabling transparent diagnosis and targeted adjustment of ranking policies.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Ranking algorithms play a pivotal role in decision-making processes across diverse domains, from search engines to job applications. When rankings directly impact individuals, ensuring fairness becomes essential, particularly for groups that are marginalised or misrepresented in the data. Most of the existing group fairness frameworks often rely on ensuring proportional representation of protected groups. However, these approaches face limitations in accounting for the stochastic nature of ranking processes or the finite size of candidate pools. To this end, we present hyperFA*IR, a framework for assessing and enforcing fairness in rankings drawn from a finite set of candidates. It relies on a generative process based on the hypergeometric distribution, which models real-world scenarios by sampling without replacement from fixed group sizes. This approach improves fairness assessment when top-$k$ selections are large relative to the pool or when protected groups are small. We compare our approach to the widely used binomial model, which treats each draw as independent with fixed probability, and demonstrate$-$both analytically and empirically$-$that our method more accurately reproduces the statistical properties of sampling from a finite population. To operationalise this framework, we propose a Monte Carlo-based algorithm that efficiently detects unfair rankings by avoiding computationally expensive parameter tuning. Finally, we adapt our generative approach to define affirmative action policies by introducing weights into the sampling process.

Problem

• Ensures fairness in rankings with finite candidate pools
• Addresses limitations of proportional representation in ranking algorithms
• Improves fairness assessment for small protected groups

Innovation

• Hypergeometric distribution models finite candidate pools
• Monte Carlo algorithm detects unfair rankings efficiently
• Weighted sampling enables affirmative action policies