Position: agentic AI orchestration should be Bayes-consistent

2026-05-01

AI Summary

This work addresses the inconsistency and irrationality in AI agent decision-making under uncertainty—such as in tool invocation, expert selection, and resource allocation—by introducing Bayesian decision theory into the agent orchestration layer rather than embedding it within large language models. The proposed approach maintains beliefs over task-relevant latent variables, updates these beliefs via Bayesian inference using observations from human–agent interactions, and selects actions through utility-aware policies, thereby establishing a practical Bayesian control paradigm. This framework significantly enhances the consistency and rationality of agent decisions. The effectiveness of calibrated belief representation and explicit utility modeling is further demonstrated through concrete design patterns and illustrative examples, which collectively yield measurable improvements in orchestration performance.

Abstract

LLMs excel at predictive tasks and complex reasoning tasks, but many high-value deployments rely on decisions under uncertainty, for example, which tool to call, which expert to consult, or how many resources to invest. While the usefulness and feasibility of Bayesian approaches remain unclear for LLM inference, this position paper argues that the control layer of an agentic AI system (that orchestrates LLMs and tools) is a clear case where Bayesian principles should shine. Bayesian decision theory provides a framework for agentic systems that can help to maintain beliefs over task-relevant latent quantities, to update these beliefs from observed agentic and human-AI interactions, and to choose actions. Making LLMs themselves explicitly Bayesian belief-updating engines remains computationally intensive and conceptually nontrivial as a general modeling target. In contrast, this paper argues that coherent decision-making requires Bayesian principles at the orchestration level of the agentic system, not necessarily the LLM agent parameters. This paper articulates practical properties for Bayesian control that fit modern agentic AI systems and human-AI collaboration, and provides concrete examples and design patterns to illustrate how calibrated beliefs and utility-aware policies can improve agentic AI orchestration.

Problem

• agentic AI
• Bayesian decision theory
• uncertainty
• orchestration
• decision-making

Innovation

• Bayesian decision theory
• agentic AI orchestration
• belief updating
• utility-aware policies
• human-AI collaboration