LREF: A Novel LLM-based Relevance Framework for E-commerce

2025-03-12

AI Summary

To address low query-item relevance prediction accuracy and inherent issues in large language models (LLMs)—including over-optimism bias and excessive recall—in e-commerce search, this paper proposes an end-to-end relevance modeling framework powered by LLMs. Our method introduces a novel three-stage collaborative optimization paradigm: (1) supervised fine-tuning guided by high-quality, domain-specific e-commerce corpora; (2) multi-chain-of-thought (Multi-CoT) reasoning augmentation to enhance logical consistency; and (3) preference alignment via direct preference optimization (DPO) to mitigate LLMs’ intrinsic biases. This approach overcomes the limitations of conventional BERT-style models in semantic depth and domain knowledge coverage. Offline evaluations demonstrate substantial improvements, while online A/B tests on a leading e-commerce platform yield +4.2% lift in click-through rate and +3.7% in conversion rate. The framework has been deployed at scale, delivering measurable business impact.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Query and product relevance prediction is a critical component for ensuring a smooth user experience in e-commerce search. Traditional studies mainly focus on BERT-based models to assess the semantic relevance between queries and products. However, the discriminative paradigm and limited knowledge capacity of these approaches restrict their ability to comprehend the relevance between queries and products fully. With the rapid advancement of Large Language Models (LLMs), recent research has begun to explore their application to industrial search systems, as LLMs provide extensive world knowledge and flexible optimization for reasoning processes. Nonetheless, directly leveraging LLMs for relevance prediction tasks introduces new challenges, including a high demand for data quality, the necessity for meticulous optimization of reasoning processes, and an optimistic bias that can result in over-recall. To overcome the above problems, this paper proposes a novel framework called the LLM-based RElevance Framework (LREF) aimed at enhancing e-commerce search relevance. The framework comprises three main stages: supervised fine-tuning (SFT) with Data Selection, Multiple Chain of Thought (Multi-CoT) tuning, and Direct Preference Optimization (DPO) for de-biasing. We evaluate the performance of the framework through a series of offline experiments on large-scale real-world datasets, as well as online A/B testing. The results indicate significant improvements in both offline and online metrics. Ultimately, the model was deployed in a well-known e-commerce application, yielding substantial commercial benefits.

Problem

• Enhance e-commerce search relevance using LLMs
• Address challenges in LLM-based relevance prediction
• Improve query-product relevance with novel framework

Innovation

• Supervised fine-tuning with data selection
• Multiple Chain of Thought tuning
• Direct Preference Optimization for de-biasing