Correctness, confidence, and context: Framing software assurance in the AI age

2026-07-06

AI Summary

This work addresses the inadequacy of traditional software engineering notions of “correctness” when applied to the statistical and probabilistic outputs of generative AI systems. It proposes a novel software assurance paradigm tailored to generative AI, expanding assurance objectives from code-level specifications to alignment with system purpose. By integrating principles from software engineering, statistical learning theory, and engineering decision frameworks, the approach constructs an algorithm-agnostic, probabilistic assurance framework. Central to this paradigm is the establishment of a systematic engineering judgment mechanism that balances correctness, confidence, and contextual relevance. The framework shifts software assurance away from formal verification toward cost-effective, multidimensional strategies, offering both conceptual foundations and practical guidance for the trustworthy deployment of generative AI and advancing software engineering toward the maturity of established engineering disciplines.

Abstract

Software engineering has a complicated relationship with "correctness". We recognize the challenges of full formal rigor as well as many required properties beyond functional correctness. Although we satisfice in practice, we are still stuck in the mindset that we could reason our way to correctness, if only we had enough information. Generative AI has introduced a new dimension to assurances: its foundation is statistical rather than formal. Traditional software engineering establishes confidence through rigorous reasoning, domain knowledge and expert judgment. In contrast, generative AI's results are sophisticated predictions, in Valiant's words "probably approximately correct". This inherently limits assurances about the results are to probabilistic assertions. Further, the nuances and implicit associations that guide human judgment are not accessible to its training sets, so that tacit knowledge cannot be incorporated in its models. We have many approaches for developing assurances that a software system does what it's expected to do, though most of them focus on the specification of the code rather than the requirements for the system, let alone fitness for purpose. We have failed to develop a systematic understanding of the relative merits of the various approaches. I hope that generative AI will finally force us to tackle this. To that end, I will challenge us to think systematically about our assurance techniques. We need ways to make informed, reasoned choices about cost-effective combinations of approaches to devel-oping confidence in our systems. We call ourselves software engineers. Let's act like engineers.

Problem

• software assurance
• generative AI
• correctness
• confidence
• fitness for purpose

Innovation

• software assurance
• generative AI
• probably approximately correct
• confidence
• fitness for purpose