Experimentation Accelerator: Interpretable Insights and Creative Recommendations for A/B Testing with Content-Aware ranking

2026-02-14

AI Summary

This work addresses the inefficiencies in online A/B testing caused by limited traffic and post-hoc analyses that lack content awareness, which hinder effective variant selection and automated insight extraction. The authors propose the first end-to-end AI-driven framework that integrates historical experiment data with content embeddings to prioritize variants, explain winning outcomes, and generate novel high-potential variants. By combining a CTR ranking model, sign-constrained sparse Lasso regression for interpretable attribution, and semantic mapping of marketing attributes, the framework enables both data-driven decision-making and creative augmentation. Evaluation on real-world customer experiments at Adobe demonstrates significant improvements in experimental efficiency, information density, and the speed of creative iteration.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Search and Optimization: Algorithm Configuration

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Modern online experimentation faces two bottlenecks: scarce traffic forces tough choices on which variants to test, and post-hoc insight extraction is manual, inconsistent, and often content-agnostic. Meanwhile, organizations underuse historical A/B results and rich content embeddings that could guide prioritization and creative iteration. We present a unified framework to (i) prioritize which variants to test, (ii) explain why winners win, and (iii) surface targeted opportunities for new, higher-potential variants. Leveraging treatment embeddings and historical outcomes, we train a CTR ranking model with fixed effects for contextual shifts that scores candidates while balancing value and content diversity. For better interpretability and understanding, we project treatments onto curated semantic marketing attributes and re-express the ranker in this space via a sign-consistent, sparse constrained Lasso, yielding per-attribute coefficients and signed contributions for visual explanations, top-k drivers, and natural-language insights. We then compute an opportunity index combining attribute importance (from the ranker) with under-expression in the current experiment to flag missing, high-impact attributes. Finally, LLMs translate ranked opportunities into concrete creative suggestions and estimate both learning and conversion potential, enabling faster, more informative, and more efficient test cycles. These components have been built into a real Adobe product, called \textit{Experimentation Accelerator}, to provide AI-based insights and opportunities to scale experimentation for customers. We provide an evaluation of the performance of the proposed framework on some real-world experiments by Adobe business customers that validate the high quality of the generation pipeline.

Problem

• A/B testing
• content-aware ranking
• experimentation bottleneck
• insight extraction
• creative iteration

Innovation

• content-aware ranking
• interpretable A/B testing
• treatment embeddings
• opportunity index
• LLM-driven recommendations