Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search

2026-06-14

AI Summary

This work addresses the challenge of automatically generating sponsored search ad copy that effectively balances user intent, world knowledge, and unique selling points. To this end, we propose Interactor, a novel framework that uniquely integrates agent-based reinforcement learning with a multi-dimensional generative reward model (GenRM). Through an iterative, multi-turn generation mechanism guided by reasoning-based feedback, Interactor optimizes ad descriptions while preserving landing page consistency. The approach significantly enhances both the knowledge richness and factual fidelity of generated ad copy. Extensive experiments on industrial-scale datasets demonstrate that Interactor substantially outperforms existing methods. The system was deployed in a major search advertising platform in May 2026, yielding measurable improvements in ad revenue and user experience.

Abstract

This paper focuses on automatically generating informative ad descriptions in sponsored search. Unlike ad titles which are usually optimized to attract user click feedbacks, ad descriptions have a longer text span and possess the potential of incorporating world knowledge to address user search intents while presenting the fine-grained selling points of the ads. We propose Interactor, a multi-turn iterative creation framework optimized with agentic RL for ad description generation. The generation model acts as a policy that interacts with a customized environment consisting of multiple generative reward models. Given initial generations by the policy, the customized GenRMs evaluate multi-dimensional qualities including knowledge capacity and landing page consistency, providing both binary signals and reasoning feedbacks. The policy then iteratively refines the descriptions based on such feedbacks to ensure continuous improvement. Experiments on industrial datasets show that the Interactor framework significantly outperforms state-of-the-art approaches in generating knowledge-rich and faithful ad descriptions. Since May 2026, it has been deployed online in a leading search ads system, contributing to both ad revenue and user experience.

Problem

• ad description generation
• sponsored search
• world knowledge
• user search intent
• landing page consistency

Innovation

• agentic RL
• iterative creation
• ad description generation
• generative reward models
• sponsored search