MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

2026-06-22

AI Summary

This work addresses the unclear efficacy and combinatorial complexity of prompt optimization in multi-agent large language model systems. For the first time, it systematically evaluates two scalable prompt optimization approaches—both natural extensions of state-of-the-art single-agent methods—across diverse tasks, workflows, communication protocols, and team sizes. The experiments demonstrate that prompt optimization can significantly enhance system performance without fine-tuning; however, its benefits are highly sensitive to system configuration. These findings delineate the boundaries of achievable gains, reveal critical sensitivities, and expose limitations of current methods, thereby establishing an empirical foundation and highlighting open directions for system-level optimization in multi-agent settings.

Abstract

Multi-agent systems (MAS) offer a scalable path forward for agentic AI, comprising multiple LLM-based agents, each assigned a system prompt and a position within a workflow that governs inter-agent coordination and output aggregation. System prompts thus form a critical and accessible optimization surface: they specify agents' roles and behaviors, enabling system-level improvements without model finetuning. Although prompt optimization has shown substantial potential for single LLMs, extending it to MAS poses distinct challenges, notably an exponentially growing search space. It remains unclear whether, when, and by how much prompt optimization improves MAS performance, and how sensitive such gains are to system configuration. In this work, we systematically study system-prompt optimization across a broad range of MAS setups varying in task, workflow, communication protocol, and team size, benchmarking two prompt optimizers that naturally extend state-of-the-art single-agent methods. The results reveal its potential to unlock significant gains while exposing open challenges, characterizing when and how much prompt optimization helps across diverse MAS settings.

Problem

• multi-agent systems
• prompt optimization
• large language models
• system prompts
• performance gains

Innovation

• multi-agent systems
• prompt optimization
• system prompts
• LLM coordination
• MAS-PromptBench