Managing Uncertainty in LLM-based Multi-Agent System Operation

2026-02-26

AI Summary

This work addresses the propagation of uncertainty in LLM-driven multi-agent systems within safety-critical applications such as cardiac ultrasound diagnosis, where uncertainties arising from coordination, data flow, human–agent interaction, and control logic cannot be mitigated by model accuracy alone. For the first time, uncertainty is treated as a first-class concern in software engineering, and a full lifecycle management paradigm is proposed for LLM-based multi-agent systems. The approach distinguishes between epistemic and ontological uncertainty and establishes a structured governance framework encompassing representation, identification, evolution, and adaptation. Integrating uncertainty taxonomy theory, multi-agent architecture, and runtime adaptation mechanisms, the framework significantly enhances the reliability and diagnosability of diagnostic reasoning in real-world clinical deployments, enabling safe and controllable operation.

Technology Category

• Multiagent Systems: Multiagent Systems under Uncertainty
• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Applying LLM-based multi-agent software systems in safety-critical domains such as lifespan echocardiography introduces system-level risks that cannot be addressed by improving model accuracy alone. During system operation, beyond individual LLM behavior, uncertainty propagates through agent coordination, data pipelines, human-in-the-loop interaction, and runtime control logic. Yet existing work largely treats uncertainty at the model level rather than as a first-class software engineering concern. This paper approaches uncertainty from both system-level and runtime perspectives. We first differentiate epistemological and ontological uncertainties in the context of LLM-based multi-agent software system operation. Building on this foundation, we propose a lifecycle-based uncertainty management framework comprising four mechanisms: representation, identification, evolution, and adaptation. The uncertainty lifecycle governs how uncertainties emerge, transform, and are mitigated across architectural layers and execution phases, enabling structured runtime governance and controlled adaptation. We demonstrate the feasibility of the framework using a real-world LLM-based multi-agent echocardiographic software system developed in clinical collaboration, showing improved reliability and diagnosability in diagnostic reasoning. The proposed approach generalizes to other safety-critical LLM-based multi-agent software systems, supporting principled operational control and runtime assurance beyond model-centric methods.

Problem

• uncertainty management
• LLM-based multi-agent systems
• safety-critical systems
• runtime uncertainty
• software engineering

Innovation

• uncertainty management
• LLM-based multi-agent systems
• runtime governance
• safety-critical software
• uncertainty lifecycle