Data-Driven Persona-Conditioned Agents for A/B Test Simulation

2026-09-01

AI Summary

本文提出一种基于数据驱动的虚拟人物代理框架，利用LLM预测A/B测试结果，减少实际用户流量需求和工程努力，提高实验预筛选效率。

Abstract

A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns, engagement signals, and inferred demographics-enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

Problem

• A/B testing
• simulation framework
• data-driven personas

Innovation

• data-driven personas
• A/B test simulation
• LLM-powered agents
• behavioral signals