MetaSynth: Multi-Agent Metadata Generation from Implicit Feedback in Black-Box Systems

2025-10-01

AI Summary

To address challenges in metadata (title/description) optimization for search and recommendation systems—including black-box ranking functions, absence of explicit supervision labels, and delayed implicit feedback—this paper proposes MetaSynth, a multi-agent retrieval-augmented framework. MetaSynth is the first approach to directly learn metadata generation from post-hoc implicit signals (e.g., click-through rate). It constructs a high-ranking historical example repository and employs an iterative evaluator-generator agent loop, integrating relevance, generalizability, and compliance constraints within a RAG-LLM architecture for candidate snippet generation and refinement. Experiments on proprietary e-commerce data and Amazon Reviews demonstrate significant improvements over strong baselines, with notable gains in NDCG and MRR. Large-scale online A/B testing confirms a 10.26% lift in click-through rate and a 7.51% increase in total clicks.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Meta titles and descriptions strongly shape engagement in search and recommendation platforms, yet optimizing them remains challenging. Search engine ranking models are black box environments, explicit labels are unavailable, and feedback such as click-through rate (CTR) arrives only post-deployment. Existing template, LLM, and retrieval-augmented approaches either lack diversity, hallucinate attributes, or ignore whether candidate phrasing has historically succeeded in ranking. This leaves a gap in directly leveraging implicit signals from observable outcomes. We introduce MetaSynth, a multi-agent retrieval-augmented generation framework that learns from implicit search feedback. MetaSynth builds an exemplar library from top-ranked results, generates candidate snippets conditioned on both product content and exemplars, and iteratively refines outputs via evaluator-generator loops that enforce relevance, promotional strength, and compliance. On both proprietary e-commerce data and the Amazon Reviews corpus, MetaSynth outperforms strong baselines across NDCG, MRR, and rank metrics. Large-scale A/B tests further demonstrate 10.26% CTR and 7.51% clicks. Beyond metadata, this work contributes a general paradigm for optimizing content in black-box systems using implicit signals.

Problem

• Optimizing meta titles and descriptions without explicit labels
• Leveraging implicit feedback from black-box ranking systems
• Generating diverse and historically successful metadata candidates

Innovation

• Multi-agent framework learns from implicit search feedback
• Builds exemplar library from top-ranked historical results
• Iteratively refines outputs via evaluator-generator feedback loops