GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents

2025-11-02

AI Summary

Online A/B testing is costly, risky, and time-consuming; offline policy evaluation (OPE) mitigates these issues but suffers from reliability degradation due to poor implementation quality—yet no prior work explores LLM-based agents for automated OPE code optimization. This paper introduces GrowthHacker, the first benchmark for evaluating coding LLM agents in OPE, and proposes a two_agent architecture to systematically study agent-driven OPE code generation, iterative refinement, and closed-loop evaluation. Leveraging real-world datasets built upon Open Bandit Pipeline and Scope-RL, and integrating frameworks including CrewAI and AutoGen, we conduct comprehensive comparative experiments. Results show that two_agent achieves a 106.7% average improvement on positive metrics, 100% reliability, and 45% success rate—substantially outperforming all baselines. This validates LLM agents as effective and feasible “growth hackers” for automating and enhancing OPE robustness and efficiency.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

With the software industry shifting toward a data-driven culture, online A/B testing is a key tool for evaluating new technologies. However, deploying such experiments requires substantial resources, may negatively impact users, and involves long data collection periods. To address this, extit{off-policy evaluation (OPE)}, or offline A/B testing, uses logged data to assess technologies and is fundamental in Reinforcement Learning, making it crucial in domains where online testing is costly or risky, such as healthcare, recommender systems, education, dialog systems, and robotics. Despite advances in coding LLMs and agentic AI, little is known about leveraging them to optimize OPE results. We investigate whether LLMs and LLM-based agents can improve OPE performance via code optimization. We propose extit{GrowthHacker}, a benchmark with agent and baseline methods on large-scale real-world datasets, which iteratively optimizes code, evaluates results, and begins new optimization cycles. We collected datasets, established protocols, implemented baselines for OPE on the Open Bandit Pipeline (OBP)~cite{saito2021openbanditdatasetpipeline} and Scope-RL~cite{kiyohara2023scope}, and developed the extit{two_agent} framework, which reduces system complexity while preserving optimization effectiveness. Results show the two_agent framework achieves 100% reliability and the highest average improvement of 106.7% among positive outcomes. Both two_agent and CrewAI reach 45% success rates, outperforming AutoGen's 34%. These findings demonstrate the feasibility of LLM-based agents as automated "growth hackers" to enhance OPE systems, with implications for scaling data-driven decision-making in production.

Problem

• Optimizing off-policy evaluation through code-modifying LLM agents
• Reducing reliance on costly online A/B testing in critical domains
• Automating iterative code optimization cycles for improved OPE performance

Innovation

• Uses code-modifying LLM agents for optimization
• Implements two_agent framework to reduce complexity
• Iteratively optimizes and evaluates OPE performance