LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBay

2025-08-05

AI Summary

To address low relevance in keyword recommendations for eBay advertising—causing search congestion and degraded seller experience—this paper proposes a two-stage LLM knowledge distillation framework. First, an LLM-as-a-judge generates high-quality pseudo-labels, augmented by a cross-encoder that explicitly models seller judgment logic. Second, multi-task knowledge distillation transfers this refined relevance signal to an efficient dual-encoder architecture, explicitly disentangling and correcting selection bias inherent in click data. This work is the first to synergistically integrate LLM-as-a-judge with cross-encoder–assisted distillation for advertising keyword recommendation. It preserves retrieval efficiency while substantially improving relevance: offline evaluation shows a 12.3% gain in NDCG@10; online A/B tests confirm concurrent increases in ad CTR and seller adoption rate, enhancing both intent alignment and business impact.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Recommender Systems
• Machine Learning: Learning Preferences or Rankings

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Ad search and search for Web retail

Abstract

Sellers at eBay are recommended keyphrases to bid on to enhance the performance of their advertising campaigns. The relevance of these keyphrases is crucial in avoiding the overcrowding of search systems with irrelevant items and maintaining a positive seller perception. It is essential that keyphrase recommendations align with both seller and Search judgments regarding auctions. Due to the difficulty in procuring negative human judgment at scale, employing LLM-as-a-judge to mimic seller judgment has been established as the norm in several studies. This study introduces a novel two-step LLM distillation process from a LLM-judge used to debias our Embedding Based Retrieval (EBR) model from the various biases that exist in click-data. We distill from an LLM teacher via a cross-encoder assistant into a bi-encoder student using a multi-task training approach, ultimately employing the student bi-encoder to retrieve relevant advertiser keyphrases. We show that integrating a knowledge distillation process from LLMs in a multi-task training setup enhances bi-encoder performance in retrieving relevant advertiser keyphrases at eBay.

Problem

• Improving relevance of advertiser keyphrase recommendations at eBay
• Debiasing EBR models using LLM distillation from click-data biases
• Enhancing bi-encoder performance via multi-task LLM knowledge distillation

Innovation

• Two-step LLM distillation process
• Cross-encoder assistant for debiasing
• Multi-task training for bi-encoder