GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation

2025-10-08

AI Summary

Text-to-image generation often suffers from semantic inconsistency and missing details when processing complex, lengthy prompts. Existing approaches either require model fine-tuning or lack systematic error analysis and interpretable optimization mechanisms. This paper proposes a model-agnostic, interpretable test-time prompt optimization framework grounded in multi-agent collaboration. It integrates automated error diagnosis, clustering-driven adaptive exploration, fine-grained verification, and memory-augmented refinement to dynamically and iteratively correct input prompts. Crucially, no model fine-tuning is required—only prompt engineering and inference-time optimization are leveraged to significantly enhance generation quality. On DPG-bench and Geneval benchmarks, the method improves text–image alignment by 16.9% and 5.7%, respectively, while simultaneously boosting structural coherence and detail fidelity.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Text-to-image synthesis has made remarkable progress, yet accurately interpreting complex and lengthy prompts remains challenging, often resulting in semantic inconsistencies and missing details. Existing solutions, such as fine-tuning, are model-specific and require training, while prior automatic prompt optimization (APO) approaches typically lack systematic error analysis and refinement strategies, resulting in limited reliability and effectiveness. Meanwhile, test-time scaling methods operate on fixed prompts and on noise or sample numbers, limiting their interpretability and adaptability. To solve these, we introduce a flexible and efficient test-time prompt optimization strategy that operates directly on the input text. We propose a plug-and-play multi-agent system called GenPilot, integrating error analysis, clustering-based adaptive exploration, fine-grained verification, and a memory module for iterative optimization. Our approach is model-agnostic, interpretable, and well-suited for handling long and complex prompts. Simultaneously, we summarize the common patterns of errors and the refinement strategy, offering more experience and encouraging further exploration. Experiments on DPG-bench and Geneval with improvements of up to 16.9% and 5.7% demonstrate the strong capability of our methods in enhancing the text and image consistency and structural coherence of generated images, revealing the effectiveness of our test-time prompt optimization strategy. The code is available at https://github.com/27yw/GenPilot.

Problem

• Optimizing text prompts to improve image generation accuracy
• Addressing semantic inconsistencies in complex text-to-image synthesis
• Developing model-agnostic prompt refinement without training requirements

Innovation

• Multi-agent system optimizes prompts at test time
• Plug-and-play approach analyzes errors and refines iteratively
• Model-agnostic method enhances text-image consistency directly