Improving Generative Ad Text on Facebook using Reinforcement Learning

2025-07-29

AI Summary

This study investigates how reinforcement learning (RL) can be applied to post-train large language models (LLMs) to improve Facebook ad copy generation and quantify its economic impact. We propose RLPF (Reinforcement Learning with Performance Feedback), an RL-based fine-tuning method that leverages sparse, real-world business metrics—such as historical click-through rate (CTR)—as reward signals for end-to-end ad copy optimization. To our knowledge, this is the first large-scale industrial validation demonstrating measurable improvements in actual business KPIs from RL-driven generative AI: in an A/B test spanning nearly 35,000 advertisers, RLPF increased CTR by 6.7%, significantly improved return on investment (ROI), and enhanced creative diversity. Our core contribution is a lightweight, feedback-driven RL post-training paradigm tailored to production environments, along with the first systematic quantification of RL’s economic value in deploying generative AI commercially.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: Generation
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Generative artificial intelligence (AI), in particular large language models (LLMs), is poised to drive transformative economic change. LLMs are pre-trained on vast text data to learn general language patterns, but a subsequent post-training phase is critical to align them for specific real-world tasks. Reinforcement learning (RL) is the leading post-training technique, yet its economic impact remains largely underexplored and unquantified. We examine this question through the lens of the first deployment of an RL-trained LLM for generative advertising on Facebook. Integrated into Meta's Text Generation feature, our model, "AdLlama," powers an AI tool that helps advertisers create new variations of human-written ad text. To train this model, we introduce reinforcement learning with performance feedback (RLPF), a post-training method that uses historical ad performance data as a reward signal. In a large-scale 10-week A/B test on Facebook spanning nearly 35,000 advertisers and 640,000 ad variations, we find that AdLlama improves click-through rates by 6.7% (p=0.0296) compared to a supervised imitation model trained on curated ads. This represents a substantial improvement in advertiser return on investment on Facebook. We also find that advertisers who used AdLlama generated more ad variations, indicating higher satisfaction with the model's outputs. To our knowledge, this is the largest study to date on the use of generative AI in an ecologically valid setting, offering an important data point quantifying the tangible impact of RL post-training. Furthermore, the results show that RLPF is a promising and generalizable approach for metric-driven post-training that bridges the gap between highly capable language models and tangible outcomes.

Problem

• Aligning LLMs for real-world ad text generation tasks
• Quantifying economic impact of RL post-training on ad performance
• Bridging gap between language models and tangible advertising outcomes

Innovation

• Reinforcement learning with performance feedback (RLPF)
• AdLlama model for generative ad text
• Large-scale A/B testing on Facebook