Prompt Optimization via Retrieved Reasoning Assets and Multi-Agent Analysis

2025-10-18

AI Summary

Existing prompt optimization methods treat evaluation as a black box, relying solely on numerical feedback without providing interpretable diagnostics for failure causes, resulting in opaque, unauditible, and uncontrollable optimization. To address this, we propose MA-SAPO—a Multi-Agent Scoring-Aware Prompt Optimization framework—that maps evaluation scores to specific prompt deficiencies via structured reasoning chains and generates evidence-backed revision suggestions. Its key innovation is a reusable reasoning asset mechanism, integrating retrieval augmentation with multi-agent collaborative diagnosis to enable transparent, auditable, and controllable iterative optimization. On the HelpSteer1/2 benchmarks, MA-SAPO significantly outperforms single-prompt baselines, retrieval-augmented approaches, and state-of-the-art multi-agent methods, demonstrating systematic improvement and strong generalization.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Prompt Engineering / Prompting
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Prompt optimization has emerged as an effective alternative to retraining for improving the performance of Large Language Models (LLMs). However, most existing approaches treat evaluation as a black box, relying solely on numerical scores while offering limited insight into why a prompt succeeds or fails. They also depend heavily on trial-and-error refinements, which are difficult to interpret and control. In this paper, we introduce MA-SAPO, a Multi-Agent framework for Score-Aware Prompt Optimization. Compared to prior methods, MA-SAPO explicitly couples evaluation outcomes with structured reasoning to guide systematic edits. The framework specifically consists of two stages: during the Reasoning Phase, agents collaboratively explain metric scores, diagnose weaknesses, and synthesize targeted refinements that are stored as reusable reasoning assets; during the Test Phase, agents retrieve these assets to analyze optimized prompts and apply only evidence-grounded edits. By turning evaluation signals into interpretable reasoning chains, MA-SAPO produces prompt refinements that are more transparent, auditable, and controllable. Experiments on the HelpSteer1/2 benchmarks demonstrate consistent improvements over single-pass prompting, retrieval-augmented baselines, and prior multi-agent strategies, validating the effectiveness of our approach.

Problem

• Optimizing prompts through reusable reasoning assets and multi-agent analysis
• Providing transparent insights into prompt success and failure reasons
• Systematically guiding prompt edits using evidence-grounded evaluation signals

Innovation

• Multi-agent framework for score-aware prompt optimization
• Retrieves reusable reasoning assets to guide edits
• Turns evaluation signals into interpretable reasoning chains