Automating Personalization: Prompt Optimization for Recommendation Reranking

2025-04-04

AI Summary

To address the scalability limitations of manual prompting and the interference of unstructured item metadata in LLM-based recommendation re-ranking, this paper proposes the AGP framework. Methodologically, AGP (1) introduces a position-aware feedback mechanism for fine-grained, sequence-sensitive ranking refinement; (2) establishes an automated prompt optimization pipeline integrating user profile generation, feedback-driven fine-tuning, and batched gradient aggregation—replacing labor-intensive prompt engineering; and (3) enhances model generalization via batched feedback training. Experiments across multiple public benchmarks demonstrate an average 12.7% improvement in NDCG@10 and a 90% reduction in prompt generation overhead. The core contribution lies in being the first to incorporate both position-aware feedback and batched gradient aggregation into LLM-based re-ranking, enabling efficient, scalable, and personalized prompt adaptation.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Modern recommender systems increasingly leverage large language models (LLMs) for reranking to improve personalization. However, existing approaches face two key limitations: (1) heavy reliance on manually crafted prompts that are difficult to scale, and (2) inadequate handling of unstructured item metadata that complicates preference inference. We present AGP (Auto-Guided Prompt Refinement), a novel framework that automatically optimizes user profile generation prompts for personalized reranking. AGP introduces two key innovations: (1) position-aware feedback mechanisms for precise ranking correction, and (2) batched training with aggregated feedback to enhance generalization.

Problem

• Automating prompt optimization for personalized recommendation reranking
• Reducing reliance on manual prompts in LLM-based recommender systems
• Improving handling of unstructured metadata for better preference inference

Innovation

• Automated prompt refinement for personalization
• Position-aware feedback for ranking correction
• Batched training with aggregated feedback