Sequential Fairness Auditing with Limited Output Access

2026-06-29

AI Summary

This work addresses the challenge of auditing external fairness in real-world AI systems, where auditors typically have only limited query access to model outputs. Existing approaches rely on static datasets and are ill-suited for sequential auditing scenarios. To bridge this gap, the paper introduces the first sequential fairness auditing framework tailored to practical deployment constraints, formulating the audit as a tolerance-aware sequential hypothesis test. Built upon generalized likelihood ratio methods, the framework accommodates diverse model outputs—including decisions, scores, and logits—and dynamically accumulates evidence, enabling early termination once a conclusive determination is reached. Empirical results demonstrate that both the type of fairness metric and the richness of model output significantly influence auditing efficiency: richer outputs generally reduce query complexity substantially, though their advantage diminishes when the system’s fairness level is near the decision threshold.

Abstract

External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited access to deployed models and must rely on query-based interactions. Most existing fairness evaluation methods assume static datasets and fixed-sample statistical tests, making them poorly suited to real-world auditing scenarios in which evidence must be collected sequentially under query constraints. In this work, we formulate fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access. We develop a sequential generalized likelihood-ratio framework that allows auditors to accumulate evidence from a finite audit pool and stop once sufficient support for compliance or violation has been obtained. The framework is instantiated for decision-based Statistical Parity and Equal Opportunity audits, and extended to score- and logit-based proxy audits when richer observables are available. Our results show that both the fairness metric and the level of model access significantly affect audit efficiency, and that the benefits of richer output information are not uniform across auditing settings. In particular, richer outputs can substantially reduce the number of queries required for some fairness metrics and operating regimes, while offering limited gains in near-threshold cases. This work provides a practical statistical framework for sequential fairness auditing under realistic deployment constraints.

Problem

• fairness auditing
• sequential hypothesis testing
• limited model access
• query-based evaluation
• AI governance

Innovation

• sequential auditing
• fairness evaluation
• limited model access
• generalized likelihood ratio
• adaptive hypothesis testing