CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

2024-12-02

arXiv.org

AI Summary

Commercial search faces critical challenges in query-item relevance modeling, including insufficient domain knowledge, underutilized in-context learning, inadequate integration of multi-field item text (e.g., title, attributes, description), and lack of background knowledge. Method: This paper proposes CPRM, a continual pre-training framework that introduces a novel context-aware pre-training paradigm—jointly modeling user queries with heterogeneous item textual fields—and employs an item reading comprehension task to automatically generate structured summaries and associated queries, thereby dynamically enriching domain knowledge. Contribution/Results: Offline experiments and online A/B tests demonstrate that CPRM significantly outperforms strong baselines across core metrics—including relevance ranking quality, click-through rate (CTR), and conversion rate (CVR)—validating its effectiveness and practicality in industrial-scale search systems.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

Problem

• Enhance domain-specific knowledge in LLMs
• Improve in-context learning for relevance modeling
• Utilize structured item text effectively

Innovation

• Joint pre-training with queries and items
• In-context pre-training for sequence learning
• Reading comprehension for domain knowledge