From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift

2025-09-30

AI Summary

Group fairness metrics (e.g., equal opportunity) exhibit high sensitivity to distributional shifts, undermining the reliability of fairness audits. To address this, we propose a Wasserstein distributionally robust fairness auditing framework. We introduce ε-Wasserstein distribution fairness (ε-WDF), a unified fairness notion that subsumes multiple group fairness criteria and provides provable worst-case fairness certification under distributional uncertainty, along with finite-sample guarantees. Leveraging Wasserstein robust optimization, strong duality theory, and functional modeling of conditional probabilities, we design DRUNE—a computationally efficient estimator for ε-WDF. Extensive experiments across standard benchmarks and diverse classifiers demonstrate that ε-WDF significantly enhances the stability and robustness of fairness evaluation under distribution shift, enabling reliable, certifiably robust out-of-distribution fairness auditing.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Group-fairness metrics (e.g., equalized odds) can vary sharply across resamples and are especially brittle under distribution shift, undermining reliable audits. We propose a Wasserstein distributionally robust framework that certifies worst-case group fairness over a ball of plausible test distributions centered at the empirical law. Our formulation unifies common group fairness notions via a generic conditional-probability functional and defines $varepsilon$-Wasserstein Distributional Fairness ($varepsilon$-WDF) as the audit target. Leveraging strong duality, we derive tractable reformulations and an efficient estimator (DRUNE) for $varepsilon$-WDF. We prove feasibility and consistency and establish finite-sample certification guarantees for auditing fairness, along with quantitative bounds under smoothness and margin conditions. Across standard benchmarks and classifiers, $varepsilon$-WDF delivers stable fairness assessments under distribution shift, providing a principled basis for auditing and certifying group fairness beyond observational data.

Problem

• Certifying worst-case group fairness under distribution shifts
• Developing robust framework for stable fairness audits
• Addressing brittleness of fairness metrics across distribution shifts

Innovation

• Wasserstein distributionally robust framework certifies worst-case group fairness
• Unifies fairness notions via generic conditional-probability functional
• Provides tractable estimator with finite-sample certification guarantees