UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

2026-07-02

AI Summary

This work addresses the issue of hallucination propagation in existing large language model–driven multi-agent software development frameworks, which often stems from neglecting the reliability of intermediate outputs and ultimately degrades software quality. To mitigate this, the authors propose an uncertainty-aware multi-agent collaboration framework that quantifies response uncertainty through lightweight token-level log-probability estimation. By integrating phase-adaptive threshold calibration, the framework selectively triggers retrieval-augmented verification at high-risk stages, enabling reliable and context-sensitive collaboration. Evaluated on the SRDD benchmark, the proposed approach significantly outperforms current single- and multi-agent methods across key metrics, including completeness, executability, consistency, and overall software quality.

Abstract

Software development is a complex task that demands cooperation among agents with diverse roles. Large language models (LLMs) have enabled autonomous multi-agent software development frameworks that leverage role-based collaboration to automate requirements analysis, coding, testing, and refinement. However, existing approaches typically assume that intermediate agent outputs are equally reliable, leaving them vulnerable to hallucination propagation, where incorrect decisions generated in early development phases are transferred to downstream agents and negatively impact final software quality. To address this challenge, we propose UA-ChatDev, an uncertainty-aware multi-agent software development framework that integrates uncertainty quantification into agent interactions. It introduces a lightweight uncertainty estimation mechanism based on token-level log probabilities to assess the confidence of agent responses and employs phase-aware threshold calibration to selectively trigger retrieval-based verification when uncertainty exceeds acceptable levels. Extensive experiments on the SRDD benchmark demonstrate that UA-ChatDev consistently outperforms existing single-agent and multi-agent software development frameworks across completeness, executability, consistency, and overall quality metrics. Further ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.

Problem

• multi-agent collaboration
• hallucination propagation
• software development
• uncertainty awareness
• LLM reliability

Innovation

• uncertainty-aware
• multi-agent collaboration
• hallucination mitigation
• token-level uncertainty estimation
• retrieval-based verification