Semantic Uncertainty-Guided Orchestration in Hierarchical Multi-Agent Systems

2026-08-10

AI Summary

This study addresses the issue of error propagation in multi-agent systems under uncertainty caused by the lack of reasoning reliability assessment. To mitigate this, we propose HAS-SUM, a semantic uncertainty-guided orchestration framework. This approach innovatively introduces architecture-agnostic semantic entropy and density metrics to quantify trustworthiness in intermediate reasoning, thereby enabling adaptive verification and response selection. Experimental results on benchmarks such as StrategyQA demonstrate that HAS-SUM significantly enhances both reliability and hallucination resistance in complex reasoning tasks. Consequently, this work establishes an effective semantic-level orchestration paradigm for constructing robust multi-agent systems capable of maintaining performance integrity despite inherent uncertainties.

Abstract

As large language model (LLM)-based multi-agent systems become increasingly capable, coordinating agents under uncertainty becomes a fundamental challenge. Existing orchestration strategies typically rely on fixed interaction patterns and often lack mechanisms for assessing the reliability of intermediate reasoning steps, allowing errors and hallucinations to propagate through the system. This paper introduces a semantic-uncertainty-guided orchestration approach, HASSUM as a general framework for uncertainty-aware coordination in multi-agent systems. The method estimates uncertainty using semantic entropy and semantic density, which measure trust at the level of answer semantics rather than output probabilities. These signals enable adaptive orchestration decisions, including output verification, selective reprompting, additional deliberation, and confidence-aware response selection. Because the approach operates independently of any particular agent architecture, it can be integrated into a broad range of hierarchical and collaborative multi-agent systems. The evaluations demonstrate an implementation within a hierarchical agent framework and evaluate it on StrategyQA, JailbreakBench, and TruthfulQA benchmarks. Across tasks that require complex reasoning and are prone to ambiguity or hallucinations, uncertainty-guided orchestration yields more reliable outcomes than uncertainty-unaware coordination. Semantic entropy and semantic density in tandem outperformed either metric alone. Ablations testing different thresholds and model sizes demonstrated that both influence the effectiveness of semantic metrics. The results suggest that semantic uncertainty is a practical and general-purpose signal for improving robustness and trustworthiness in agentic AI systems.

Problem

• Multi-Agent Systems
• Semantic Uncertainty
• Orchestration
• Hallucination
• Reliability

Innovation

• Semantic Uncertainty
• Multi-Agent Orchestration
• Semantic Entropy
• Semantic Density
• Adaptive Coordination