Scaling Dense Retrieval with LLM-Annotated Training Data: Structured Mining and Progressive Curriculum for E-Commerce Sponsored Search

2026-06-22

AI Summary

This work addresses position bias in click signals and data sparsity for long-tail queries in e-commerce ad retrieval by proposing a self-supervised dense retrieval training framework that requires no manual annotations. Structured training signals are derived from output discrepancies among heterogeneous retrieval channels, while relevance labels are generated via a high-agreement cascade of three large language models. A progressive curriculum learning strategy is employed, spanning 240 million samples across five difficulty levels. Deployed in Walmart’s ad search system, the approach improves NDCG@10 by 5.1%—with pronounced gains on long-tail queries—and reduces low-quality results from 8.7% to 3.5%. A/B testing further demonstrates increases of 2.80% in ad spend, 1.4% in CTR, 2.8% in eCPM, and 2.9% in click-through conversion rate.

Abstract

How can we generate high-quality training data for dense retrieval models at production scale, without relying on click signals or manual annotation? This question is critical for e-commerce sponsored search, where click-based training suffers from position bias and tail-query sparsity, and manual labeling at the scale of hundreds of millions of query-item pairs is economically infeasible. Our work is driven by the following insight: heterogeneous retrieval systems disagree on most items they retrieve, and this disagreement creates a natural source of structured training signal -- easy positives where all systems agree, hard positives that only lexical systems find, and hard negatives that fool exactly one system. As our key novelty, we combine three ideas into an end-to-end pipeline: (a) multi-channel retrieval mining with rank metadata from three production systems, (b) graded-relevance annotation by a calibrated three-model cascade ) that reaches 89.1% agreement with trained human annotators, and (c) three-stage progressive curriculum training that organizes 240M+ training examples across five difficulty levels. We deploy the trained two-tower BERT model on Walmart's sponsored search and evaluate it against 30K queries labeled by trained third-party human annotators. First, we show that the system achieves +5.1% NDCG@10 over the click-trained production baseline, with the largest gain on tail queries . Second, we show that embarrassing retrievals (rating 0) drop from 8.7% to 3.5%. Third, a two-week online A/B test with tens of millions of ad requests per arm confirms +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate. Overall, our work provides a practical and scalable blueprint for replacing click-based training with structured LLM-annotated supervision in production retrieval systems.

Problem

• dense retrieval
• training data generation
• e-commerce sponsored search
• click bias
• long-tail queries

Innovation

• dense retrieval
• LLM-annotated data
• structured mining
• progressive curriculum learning
• e-commerce sponsored search