From Craft to Constitution: A Governance-First Paradigm for Principled Agent Engineering

2025-10-12

AI Summary

Current large language model (LLM) agents face a “craftsmanship crisis” when transitioning from prototypes to production deployment: rigidly imposing deterministic software engineering paradigms results in fragile, unpredictable, and untrustworthy systems. To address the inherent probabilistic nature of LLMs, this paper introduces a *Governance-First* paradigm. Its core innovation is ArbiterOS—a formal runtime governance system integrating principle-driven control, probabilistic reasoning modeling, and verifiable coordination mechanisms. Moving beyond imperative development, ArbiterOS ensures agent behavior that is consistent, interpretable, and controllable. Experimental evaluation demonstrates substantial improvements in stability and trustworthiness on complex tasks. ArbiterOS thus provides the first verifiable and scalable governance infrastructure for engineering LLM agents in high-stakes operational environments.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The advent of powerful Large Language Models (LLMs) has ushered in an ``Age of the Agent,'' enabling autonomous systems to tackle complex goals. However, the transition from prototype to production is hindered by a pervasive ``crisis of craft,'' resulting in agents that are brittle, unpredictable, and ultimately untrustworthy in mission-critical applications. This paper argues this crisis stems from a fundamental paradigm mismatch -- attempting to command inherently probabilistic processors with the deterministic mental models of traditional software engineering. To solve this crisis, we introduce a governance-first paradigm for principled agent engineering, embodied in a formal architecture we call ArbiterOS.

Problem

• Addressing brittle unpredictable agents in mission-critical applications
• Solving paradigm mismatch between probabilistic AI and deterministic engineering
• Introducing governance-first architecture for trustworthy autonomous systems

Innovation

• Governance-first paradigm for principled agent engineering
• Formal architecture called ArbiterOS implementation
• Addresses probabilistic processors with deterministic models