Verified Multi-Agent Orchestration: A Plan-Execute-Verify-Replan Framework for Complex Query Resolution

2026-03-11

AI Summary

This work addresses the challenges of incomplete responses and low-quality information sources commonly encountered when answering complex queries in multi-agent systems. The authors propose a closed-loop framework that orchestrates multiple specialized large language model agents through a dependency-aware sub-question directed acyclic graph for parallel execution. To ensure response quality while controlling resource consumption, the framework incorporates a validator-based adaptive replanning mechanism and configurable stopping criteria. Experimental evaluation on 25 expert-level market research queries demonstrates significant improvements over single-agent baselines, with response completeness increasing from 3.1 to 4.2 and information source quality rising from 2.6 to 4.1 on a 5-point scale.

Technology Category

• Multiagent Systems: Multiagent Planning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present Verified Multi-Agent Orchestration (VMAO), a framework that coordinates specialized LLM-based agents through a verification-driven iterative loop. Given a complex query, our system decomposes it into a directed acyclic graph (DAG) of sub-questions, executes them through domain-specific agents in parallel, verifies result completeness via LLM-based evaluation, and adaptively replans to address gaps. The key contributions are: (1) dependency-aware parallel execution over a DAG of sub-questions with automatic context propagation, (2) verification-driven adaptive replanning that uses an LLM-based verifier as an orchestration-level coordination signal, and (3) configurable stop conditions that balance answer quality against resource usage. On 25 expert-curated market research queries, VMAO improves answer completeness from 3.1 to 4.2 and source quality from 2.6 to 4.1 (1-5 scale) compared to a single-agent baseline, demonstrating that orchestration-level verification is an effective mechanism for multi-agent quality assurance.

Problem

• multi-agent orchestration
• complex query resolution
• answer completeness
• source quality
• verification

Innovation

• Verified Multi-Agent Orchestration
• Plan-Execute-Verify-Replan
• DAG-based decomposition
• LLM-based verification
• adaptive replanning