Empirical Likelihood-Based Fairness Auditing: Distribution-Free Certification and Flagging

2026-01-28

AI Summary

This work addresses the challenge of systematic performance disparities exhibited by machine learning models across sensitive subgroups in high-stakes settings, where existing fairness auditing methods often rely on strong distributional assumptions or incur prohibitive computational costs. The paper introduces, for the first time, empirical likelihood into fairness auditing, proposing a nonparametric, distribution-free statistical inference framework. By constructing a discrepancy statistic that asymptotically follows a mixture of chi-squared distributions, the method enables efficient and accurate fairness certification and subgroup bias detection without resampling. Demonstrating both computational efficiency and statistical power, the approach successfully identifies intersectional biases on the COMPAS dataset, achieves coverage matching nominal confidence levels, and accelerates computation by several orders of magnitude compared to bootstrap methods, making it suitable for large-scale deployment.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Machine learning models in high-stakes applications, such as recidivism prediction and automated personnel selection, often exhibit systematic performance disparities across sensitive subpopulations, raising critical concerns regarding algorithmic bias. Fairness auditing addresses these risks through two primary functions: certification, which verifies adherence to fairness constraints; and flagging, which isolates specific demographic groups experiencing disparate treatment. However, existing auditing techniques are frequently limited by restrictive distributional assumptions or prohibitive computational overhead. We propose a novel empirical likelihood-based (EL) framework that constructs robust statistical measures for model performance disparities. Unlike traditional methods, our approach is non-parametric; the proposed disparity statistics follow asymptotically chi-square or mixed chi-square distributions, ensuring valid inference without assuming underlying data distributions. This framework uses a constrained optimization profile that admits stable numerical solutions, facilitating both large-scale certification and efficient subpopulation discovery. Empirically, the EL methods outperform bootstrap-based approaches, yielding coverage rates closer to nominal levels while reducing computational latency by several orders of magnitude. We demonstrate the practical utility of this framework on the COMPAS dataset, where it successfully flags intersectional biases, specifically identifying a significantly higher positive prediction rate for African-American males under 25 and a systemic under-prediction for Caucasian females relative to the population mean.

Problem

• fairness auditing
• algorithmic bias
• performance disparities
• distribution-free inference
• subpopulation discovery

Innovation

• empirical likelihood
• fairness auditing
• non-parametric inference
• distribution-free certification
• intersectional bias