🤖 AI Summary
To address low user engagement and poor interaction rates caused by template-based email subject lines in e-commerce marketing, this paper proposes a large language model (LLM)-driven method for generating personalized email subject lines, tightly integrating user profiling with item recommendation content to enable thematic and user-specific headline generation. This work represents the first systematic deployment of LLMs for email subject line generation at scale—across a user base exceeding one million—in a real-world e-commerce setting. We establish a comprehensive evaluation framework encompassing offline simulation, rigorous A/B testing, and online experimentation. Empirical results demonstrate statistically significant improvements in both email open rates and click-through rates. Crucially, these gains are achieved under strict safety and controllability constraints, ensuring operational stability and business impact—thereby overcoming key limitations of conventional template-driven approaches.
📝 Abstract
E-commerce marketplaces make use of a number of marketing channels like emails, push notifications, etc. to reach their users and stimulate purchases. Personalized emails especially are a popular touch point for marketers to inform users of latest items in stock, especially for those who stopped visiting the marketplace. Such emails contain personalized recommendations tailored to each user's interests, enticing users to buy relevant items. A common limitation of these emails is that the primary entry point, the title of the email, tends to follow fixed templates, failing to inspire enough interest in the contents. In this work, we explore the potential of large language models (LLMs) for generating thematic titles that reflect the personalized content of the emails. We perform offline simulations and conduct online experiments on the order of millions of users, finding our techniques useful in improving the engagement between customers and our emails. We highlight key findings and learnings as we productionize the safe and automated generation of email titles for millions of users.