Language Model Evolutionary Algorithms for Recommender Systems: Benchmarks and Algorithm Comparisons

2024-11-16

arXiv.org

AI Summary

Current prompt optimization for LLM-based recommender systems lacks multi-objective evaluation benchmarks and efficient algorithms. Method: We introduce RSBench, the first multi-objective benchmark for conversational recommendation, enabling Pareto-optimal prompt search across accuracy, diversity, and fairness; we propose three LLM-driven evolutionary algorithm frameworks—based on NSGA-II, CMA-ES, and differential evolution—that deeply integrate LLMs’ semantic understanding and generation capabilities into prompt encoding, evaluation, and crossover/mutation operations. Contribution/Results: Systematic evaluation on RSBench reveals that prompt diversity, feedback-driven assessment mechanisms, and LLM alignment are critical determinants of recommendation performance. This work establishes the first reproducible multi-objective benchmark for LLM-enhanced evolutionary optimization, introduces a novel algorithmic paradigm, and provides practical guidelines for integrating LLMs with evolutionary algorithms in recommendation.

Technology Category

• Search and Optimization: Evolutionary Computation
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Evolutionary Learning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

In the evolutionary computing community, the remarkable language-handling capabilities and reasoning power of large language models (LLMs) have significantly enhanced the functionality of evolutionary algorithms (EAs), enabling them to tackle optimization problems involving structured language or program code. Although this field is still in its early stages, its impressive potential has led to the development of various LLM-based EAs. To effectively evaluate the performance and practical applicability of these LLM-based EAs, benchmarks with real-world relevance are essential. In this paper, we focus on LLM-based recommender systems (RSs) and introduce a benchmark problem set, named RSBench, specifically designed to assess the performance of LLM-based EAs in recommendation prompt optimization. RSBench emphasizes session-based recommendations, aiming to discover a set of Pareto optimal prompts that guide the recommendation process, providing accurate, diverse, and fair recommendations. We develop three LLM-based EAs based on established EA frameworks and experimentally evaluate their performance using RSBench. Our study offers valuable insights into the application of EAs in LLM-based RSs. Additionally, we explore key components that may influence the overall performance of the RS, providing meaningful guidance for future research on the development of LLM-based EAs in RSs.

Problem

• Develop benchmarks for LLM-based evolutionary algorithms in recommender systems.
• Evaluate performance of LLM-based EAs in optimizing recommendation prompts.
• Explore key components influencing LLM-based recommender system performance.

Innovation

• LLM-based evolutionary algorithms for recommender systems
• RSBench benchmark for session-based recommendation optimization
• Pareto optimal prompts for accurate, diverse, fair recommendations