LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots

2026-05-15

AI Summary

This work addresses the challenge of integrating ad auctions into large language model (LLM)-based chatbots, where trade-offs among relevance, efficiency, and user experience are critical. Existing embedding-based retrieval methods often lead to commercial misjudgments and redundant ad insertions. To overcome these limitations, the authors propose LERA, a novel framework that leverages the semantic understanding capabilities of LLMs in the fine-ranking stage of ad auctions. LERA employs a two-stage paradigm: it first uses embeddings for coarse candidate filtering and then applies prompt engineering with logits-based scoring to generate fine-grained relevance estimates. Coupled with a threshold-based payment mechanism, LERA ensures incentive compatibility while enabling scalable, end-to-end dynamic insertion of multiple ads. Experiments on synthetic datasets demonstrate significant improvements in both ad selection accuracy and diversity, with only minimal and controllable latency overhead.

Abstract

The integration of advertising auction mechanisms into large language model (LLM)-based chatbots presents a significant opportunity for commercialization, yet poses unique challenges in balancing relevance, efficiency, and user experience. Recently, Feizi et al.~\citep{feizi2023online} and Hajiaghayi et al.~\citep{hajiaghayi2024ad} outlined a retrieve-then-generate paradigm that decouples retrieval and generation, offering lightweight ad insertion and payment determination. However, current retrieval relies solely on text embedding similarity, which may lead to commercial misinterpretation and issues such as repetitive insertions. In this paper, we propose LERA, a two-stage retrieve-then-generate auction framework tailored for LLM chatbots. In the first stage, embedding-based coarse filtering pre-selects a small set of candidate advertisers. In the second stage, the LLM itself is queried with a carefully designed prompt to produce logits over candidates, which serve as refined organic relevance scores. These scores are combined with bids, and a critical-value payment rule accounts for both the coarse-filtering and fine-ranking thresholds, ensuring truthfulness for utility-maximizing advertisers. The framework naturally extends to multiple ad insertions within dynamic dialogue flows and long responses. Experiments on a synthetic advertiser-query benchmark show that LERA substantially improves ad selection accuracy and insertion diversity while incurring only controllable latency overhead.

Problem

• ad auction
• retrieval-augmented generation
• large language models
• ad relevance
• chatbot advertising

Innovation

• LLM-enhanced RAG
• ad auction
• two-stage retrieval
• truthful mechanism
• logits-based relevance scoring