🤖 AI Summary
This work addresses the absence of continuous governance mechanisms for dynamically balancing quality, reliability, safety, latency, and cost when enterprises deploy large language model (LLM) agents. The authors propose an Evaluation-Driven Development and Operations (EDDOps) framework tailored for non-deterministic LLM agents, built upon AWS AgentCore. By integrating observability, pluggable evaluators, and an agent registry, the architecture enables evidence-driven model selection and governance throughout the agent lifecycle. This approach reframes model selection from static benchmark rankings to an economic decision grounded in cost–performance trade-offs. The efficacy of the framework is demonstrated through 30 single-turn invocations, nine multi-turn evaluations, and successful integration with a registry, collectively validating its feasibility for efficient EDDOps implementation on managed platforms.
📝 Abstract
Enterprise adoption of LLM agents requires model selection methods that balance quality, reliability, safety, latency, and cost. Evaluation-Driven Development and Operations (EDDOps) positions evaluation as a continuous governing function across the agent lifecycle rather than a terminal checkpoint. This paper presents a practitioner-oriented instantiation of EDDOps on AWS Bedrock AgentCore and proposes a cost-to-performance framework for selecting foundation models in enterprise agent architectures. We make three contributions: a conceptual synthesis explaining why traditional TDD/BDD methods are insufficient for non-deterministic LLM agents; an architectural mapping of the EDDOps reference architecture onto AgentCore Runtime, Evaluations, Agent Registry, and CloudWatch observability; and an empirical cost-to-performance decision framework validated through a proof-of-concept comparing three foundation models across two deployment paths. Using trace data from 30 single-turn invocations across six agents, 9 multi-turn evaluations, and registry-integrated governance, we show how evaluation evidence can convert model selection from a benchmark-ranking exercise into a governed economic decision. The results suggest that managed agent platforms can support EDDOps when they provide trace-native observability, pluggable evaluator frameworks, and governed registry-based discovery.