LLM Retrieval for Stable and Predictable Ad Recommendations

2026-05-20

AI Summary

This work addresses the sensitivity of traditional ad recommendation systems to minor perturbations in ad creatives, which leads to unstable predictions, poor reproducibility, and cold-start challenges. To mitigate these issues, the authors propose a semantic candidate generation framework based on fine-tuned large language models (LLMs). The approach constructs LLM-based representations of ad creatives through hierarchical semantic attribute extraction and incorporates a graph-based expansion mechanism to introduce semantically diverse variants, thereby enhancing robustness against input perturbations. The study introduces, for the first time, an evaluation protocol focused explicitly on stability and predictability, integrating fine-tuned LLMs with graph structures to achieve consistent and interpretable recommendations. Large-scale online A/B tests demonstrate that the proposed method not only significantly improves prediction stability and predictability but also enhances conventional recommendation metrics.

Abstract

Traditional ads recommendation systems have primarily focused on optimizing for prediction accuracy of click or conversion events using canonical metrics such as recall or normalized discounted cumulative gain (NDCG). With the hyper-growth of ads inventory and liquidity with generative AI technologies, the prediction stability and predictability is becoming increasingly critical. Intuitively, prediction stability and predictability can be defined to quantify system robustness with respect to minor/noisy input (ads, creatives) perturbations, the lack of which could lead to advertiser perceivable problems such as repeatability, cold start and under-exploration. In this paper, we introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system, and present an online validated semantic candidate generation framework powered by fine-tuned Large Language Models (LLMs) that showed significant improvement along these metrics by fundamentally improving the semantic-awareness of the system. The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the foundation for graph-based expansion, ensuring the retrieved candidates encapsulate semantic variants of an ad, guaranteeing that small creative variants from the advertiser yield consistent and explainable delivery results to the user. We tested this LLM ads retrieval framework in a large-scale industrial ads recommendation system, demonstrating significant improvements across offline and online A/B experiments, showcasing gains in both predictability and traditional performance metrics. Although evaluated in the ads stack, this is a general framework that can be applied broadly to any large-scale recommendation and retrieval systems facing similar scaling and predictability challenges.

Problem

• prediction stability
• predictability
• ad recommendation
• input perturbation
• system robustness

Innovation

• LLM-based retrieval
• semantic-aware recommendation
• prediction stability
• ad recommendation
• hierarchical semantic attributes