External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery

2025-06-19

AI Summary

This study evaluates the practical efficacy of Meta’s Variance Reduction System (VRS)—deployed to comply with the 2022 U.S. Department of Justice settlement agreement—in mitigating algorithmic discrimination in advertising. Using the first black-box, third-party empirical methodology designed for large-scale platforms, we conduct real-world ad delivery experiments and perform cross-group exposure and cost-attribution analysis. Results show that while VRS reduces impression variance across demographic groups, it substantially increases per-user reach costs and diminishes exposure opportunities for historically disadvantaged groups—indicating cost-shifting rather than equitable optimization. We further propose a novel, low-cost, high-coverage fair delivery mechanism that simultaneously improves exposure parity across groups and reduces advertisers’ costs. Our work provides the first empirical demonstration that fairness and commercial efficiency can be jointly enhanced—offering a more transparent, scalable, and governance-friendly alternative to current platform-level algorithmic interventions.

Technology Category

• Search and Optimization: Algorithm Configuration
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The 2022 settlement between Meta and the U.S. Department of Justice to resolve allegations of discriminatory advertising resulted is a first-of-its-kind change to Meta's ad delivery system aimed to address algorithmic discrimination in its housing ad delivery. In this work, we explore direct and indirect effects of both the settlement's choice of terms and the Variance Reduction System (VRS) implemented by Meta on the actual reduction in discrimination. We first show that the settlement terms allow for an implementation that does not meaningfully improve access to opportunities for individuals. The settlement measures impact of ad delivery in terms of impressions, instead of unique individuals reached by an ad; it allows the platform to level down access, reducing disparities by decreasing the overall access to opportunities; and it allows the platform to selectively apply VRS to only small advertisers. We then conduct experiments to evaluate VRS with real-world ads, and show that while VRS does reduce variance, it also raises advertiser costs (measured per-individuals-reached), therefore decreasing user exposure to opportunity ads for a given ad budget. VRS thus passes the cost of decreasing variance to advertisers. Finally, we explore an alternative approach to achieve the settlement goals, that is significantly more intuitive and transparent than VRS. We show our approach outperforms VRS by both increasing ad exposure for users from all groups and reducing cost to advertisers, thus demonstrating that the increase in cost to advertisers when implementing the settlement is not inevitable. Our methodologies use a black-box approach that relies on capabilities available to any regular advertiser, rather than on privileged access to data, allowing others to reproduce or extend our work.

Problem

• Evaluates Meta's ad system changes to reduce housing discrimination
• Assesses limitations of settlement terms and VRS effectiveness
• Proposes better alternative to increase ad exposure and lower costs

Innovation

• Evaluates Meta's VRS for discrimination reduction
• Proposes transparent alternative to VRS
• Uses black-box approach for reproducibility