Algorithm or Creative? A Three-Arm Experimental Design for Decomposing Algorithmic Bias in Platform A/B Tests

2026-05-22

AI Summary

Traditional two-arm A/B tests struggle to disentangle the effects of ad creatives from the confounding influence of platform algorithms in audience targeting, leading to conflation of algorithmic bias and the true impact of creative content. This study proposes a three-arm randomized controlled trial design that introduces a third group exposed only to treatment metadata while holding creatives constant. This approach enables point identification of the natural indirect effect of the algorithm and the direct effect of the creative without relying on the sequential ignorability assumption. Empirical implementation on Meta’s platform reveals that the algorithmic channel increases female exposure share by 2.07 percentage points, whereas the creative channel reduces it by 0.68 percentage points. Conventional two-arm tests substantially misestimate the algorithmic effect, exhibiting a bias as large as twofold.

Abstract

Online advertising platforms host hundreds of thousands of A/B tests, but the platform's delivery algorithm routes each creative to the audience it predicts will engage. Every two-arm test therefore conflates the creative's effect with the algorithm's targeting response, and adjusting for the realized audience is biased because audience is a post-treatment mediator. We propose a three-arm design that adds an arm exposing the algorithm to the treatment metadata while holding the user-facing creative identical to control, point-identifying the natural indirect (algorithmic) and direct (creative) effects without sequential ignorability. In a live Meta campaign with a women-targeted text fragment, the algorithmic channel raises female impression share by +2.07 ppt while the creative channel moves it by -0.68 ppt; roughly three-quarters of the absolute reallocation is algorithmic, and a conventional two-arm test understates the algorithmic channel by a factor of two. The design isolates the contribution of platform's algorithm to the outcome which is separable from creative content.

Problem

• algorithmic bias
• A/B testing
• causal inference
• online advertising
• mediation analysis

Innovation

• algorithmic bias
• three-arm design
• causal mediation
• A/B testing
• platform algorithms