Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updates

2026-01-09

arXiv.org

AI Summary

This work addresses the challenge that traditional fairness auditing methods fail under model dynamics due to changes in the model class. The authors propose a Probably Approximately Correct (PAC) auditing framework based on an Empirical Property Optimization (EPO) oracle, which efficiently estimates group fairness and other properties under arbitrary strategic model updates while preserving the audited property. The key innovation lies in introducing the SP dimension—a novel combinatorial complexity measure that captures the intricacy of permissible updates—and using it to derive distribution-free upper bounds on sample complexity. The framework naturally extends to objectives such as prediction error and robust risk, and theoretical analysis demonstrates its ability to achieve efficient estimation of statistical fairness with minimal labeled samples and strong generalization guarantees.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

As machine learning models become increasingly embedded in societal infrastructure, auditing them for bias is of growing importance. However, in real-world deployments, auditing is complicated by the fact that model owners may adaptively update their models in response to changing environments, such as financial markets. These updates can alter the underlying model class while preserving certain properties of interest, raising fundamental questions about what can be reliably audited under such shifts. In this work, we study group fairness auditing under arbitrary updates. We consider general shifts that modify the pre-audit model class while maintaining invariance of the audited property. Our goals are two-fold: (i) to characterize the information complexity of allowable updates, by identifying which strategic changes preserve the property under audit; and (ii) to efficiently estimate auditing properties, such as group fairness, using a minimal number of labeled samples. We propose a generic framework for PAC auditing based on an Empirical Property Optimization (EPO) oracle. For statistical parity, we establish distribution-free auditing bounds characterized by the SP dimension, a novel combinatorial measure that captures the complexity of admissible strategic updates. Finally, we demonstrate that our framework naturally extends to other auditing objectives, including prediction error and robust risk.

Problem

• fairness auditing
• model updates
• group fairness
• property preservation
• adaptive updates

Innovation

• PAC auditing
• Empirical Property Optimization
• SP dimension
• group fairness
• model updates