Reasonable uncertainty: Confidence intervals in empirical Bayes discrimination detection

2025-08-18

AI Summary

This paper addresses the insufficient uncertainty quantification in empirical Bayesian discrimination detection. Methodologically, it proposes a unified framework integrating sampling variability and partial identification. Its core innovation is the definition and estimation of a counterfactual odds ratio—a causal estimand—directly anchoring statistical evidence to the true magnitude of discrimination. The approach combines empirical Bayes inference, a counterfactual causal framework, and robust confidence interval construction to jointly model uncertainty arising from both the identification set and the sampling distribution. Empirical evaluation demonstrates that existing methods frequently overstate statistical significance by neglecting partial identification. In contrast, the proposed method substantially improves the accuracy of uncertainty quantification and enhances policy interpretability. It exhibits robustness and reliability in assessing discrimination intensity across multiple real-world datasets.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Causality
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

We revisit empirical Bayes discrimination detection, focusing on uncertainty arising from both partial identification and sampling variability. While prior work has mostly focused on partial identification, we find that some empirical findings are not robust to sampling uncertainty. To better connect statistical evidence to the magnitude of real-world discriminatory behavior, we propose a counterfactual odds-ratio estimand with a attractive properties and interpretation. Our analysis reveals the importance of careful attention to uncertainty quantification and downstream goals in empirical Bayes analyses.

Problem

• Address uncertainty in empirical Bayes discrimination detection
• Evaluate robustness of findings to sampling variability
• Propose counterfactual odds-ratio estimand for discrimination analysis

Innovation

• Counterfactual odds-ratio estimand for discrimination detection
• Addresses partial identification and sampling uncertainty
• Emphasizes uncertainty quantification in empirical Bayes