MARS: A Multi-Agent Framework Incorporating Socratic Guidance for Automated Prompt Optimization

2025-03-21

AI Summary

Existing automated prompt optimization methods suffer from rigid template designs and low search efficiency. To address these limitations, this paper proposes MARS, a multi-agent collaborative framework. Its core contributions are threefold: (1) seven function-specialized agents that autonomously plan and execute optimization trajectories; (2) a novel Socratic dialogue mechanism integrating Teacher–Critic–Student roles to enable interpretable, iterative, and progressive prompt refinement; and (3) a continuous modeling of the prompt space coupled with a closed-loop feedback optimization process. This paradigm transcends fixed-template constraints, exhibiting both adaptability and evolutionary capability. Extensive experiments across multiple benchmark datasets demonstrate that MARS significantly improves LLM response quality. Ablation and analytical studies further validate substantial gains in optimization efficiency, task generalization, and decision interpretability.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The basic question-answering format of large language models involves inputting a prompt and receiving a response, and the quality of the prompt directly impacts the effectiveness of the response. Automated Prompt Optimization (APO) aims to break free from the cognitive biases of manually designed prompts and explores a broader design space for prompts. However, existing APO methods suffer from limited flexibility of fixed templates and inefficient search in prompt spaces as key issues. To this end, we propose a Multi-Agent framework Incorporating Socratic guidance (MARS), which utilizes multi-agent fusion technology for automatic planning, with gradual continuous optimization and evaluation. Specifically, MARS comprises seven agents, each with distinct functionalities, which autonomously use the Planner to devise an optimization path that ensures flexibility. Additionally, it employs a Teacher-Critic-Student Socratic dialogue pattern to iteratively optimize the prompts while conducting effective search. We conduct extensive experiments on various datasets to validate the effectiveness of our method, and perform additional analytical experiments to assess the model's advancement as well as the interpretability.

Problem

• Improving prompt quality for better LLM responses
• Overcoming limitations of fixed templates in APO
• Enhancing prompt optimization via multi-agent Socratic dialogue

Innovation

• Multi-agent fusion for automatic planning
• Socratic dialogue pattern for iterative optimization
• Seven distinct agents ensuring flexibility