Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes

2024-10-30

arXiv.org

AI Summary

In auditing fairness of social media advertising algorithms, reliance on inferred demographic attributes—due to the absence of ground-truth population labels—introduces inference errors that distort bias detection. Method: This work first systematically characterizes how such inference errors propagate into fairness audit outcomes and proposes an expectation-error-correction framework to enhance audit sensitivity. It integrates error propagation analysis, a pairwise ad-audit paradigm, and bias metric correction tailored for aggregated data under label uncertainty. Results: Experiments demonstrate that the method significantly improves accuracy and sensitivity in detecting distributional bias under inferred demographics (average gain of 32.7%), effectively mitigating false negatives and false positives induced by label noise. This study establishes the first verifiable, scalable, and error-robust paradigm for black-box algorithmic fairness auditing in label-scarce settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Responsible Web: Algorithmic accountability and transparency on the web
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Auditing social-media algorithms has become a focus of public-interest research and policymaking to ensure their fairness across demographic groups such as race, age, and gender in consequential domains such as the presentation of employment opportunities. However, such demographic attributes are often unavailable to auditors and platforms. When demographics data is unavailable, auditors commonly infer them from other available information. In this work, we study the effects of inference error on auditing for bias in one prominent application: black-box audit of ad delivery using paired ads. We show that inference error, if not accounted for, causes auditing to falsely miss skew that exists. We then propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms. Our method works by adjusting for expected error due to demographic inference, and it makes skew detection more sensitive when attributes must be inferred. Because inference is increasingly used for auditing, our results provide an important addition to the auditing toolbox to promote correct audits of ad delivery algorithms for bias. While the impact of attribute inference on accuracy has been studied in other domains, our work is the first to consider it for black-box evaluation of ad delivery bias, when only aggregate data is available to the auditor.

Problem

• Auditing bias in ad delivery using inferred demographics
• Mitigating inference error effects on bias detection
• Improving skew detection in black-box ad audits

Innovation

• Uses inferred demographics for bias auditing
• Adjusts for expected demographic inference errors
• Enhances skew detection in ad delivery algorithms