Formal Security Analysis of Agent Protocol Composition

2026-06-26

AI Summary

Current AI agent protocols commonly suffer from incomplete security requirements, inconsistent enforcement, and a lack of cross-protocol accountability in both specifications and implementations. This work proposes AgentThread, a novel framework that introduces hierarchical security scopes and protocol-derived TLA+ invariants, coupled with a two-phase verification mechanism to enable end-to-end security analysis from textual specifications to runtime SDKs. By integrating formal verification, model checking, and source-code linkage techniques, AgentThread automatically compiles protocol specifications and generates replayable test cases. Evaluation across five widely used agent protocols uncovered 35 specification-level flaws, validated through 80 SDK tests, and revealed 30 previously unknown vulnerabilities that manifest only under specific protocol compositions.

Abstract

AI agent protocols define how agents use tools, delegate work, and coordinate across software systems, but their security requirements remain incomplete and inconsistently enforced across deployments. We present AgentThread, a source-linked framework for security assurance analysis of agent protocols, from specification text to running SDKs. AgentThread contributes a layered security scope, protocol-derived checks formalized as TLA+ invariants, and a two-phase checker that compiles protocol specifications into model-checkable models and replays executable counterexamples against real SDKs through protocol adapters. For each finding, AgentThread records the source text behind the check and separates violated protocol requirements from missing recommendations, hardening gaps, and unassigned cross-protocol responsibilities. Across five emerging agent protocols, AgentThread identifies 35 specification-level findings, supports them with 80 implementation tests against production SDKs and reference servers, and finds 30 additional failures that emerge only under protocol composition. We further show that only one protocol enforces a security-relevant control in practice and no protocol assigns enforcement for cross-protocol behavior. Insecurity in agent protocols is therefore not only a specification or implementation problem, but also a responsibility gap across protocols, SDKs, and deployments.

Problem

• agent protocols
• security analysis
• protocol composition
• responsibility gap
• formal verification

Innovation

• AgentThread
• formal security analysis
• protocol composition
• TLA+ invariants
• security responsibility gap