🤖 AI Summary
This work addresses key challenges enterprises face when transitioning large language model (LLM) prototypes into production—namely, insufficient auditability, unpredictable behavior, and the absence of enforceable behavioral guarantees. The authors propose “harness engineering,” a methodology that restructures prompt-driven prototypes into auditable, traceable LLM agent architectures. For the first time, enterprise-grade behavioral contracts—including entity routing, source attribution, and output formatting—are formally encoded as executable code, establishing model-agnostic, verifiable safety boundaries. Through codified contracts, runtime validators, fault-injection testing, and model-swapping evaluations on data from 25 Korean publicly listed companies, the approach demonstrates 100% compliance with specified contracts across three hosted LLMs while preserving full functional utility (120/120), significantly outperforming pure prompting or external guardrail strategies.
📝 Abstract
Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context. Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces. We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture: deterministic behavior moves into code, manifests, schemas, and validation artifacts around a replaceable composition boundary, while source-backed claims remain the authority for runtime answers. We instantiate it on a public-data slice of five Korean corporate groups (25 listed companies) and evaluate three research questions. (1) The harness preserves its source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts across the fixed validation scenarios; a fault-injection control confirms the validators flag deliberately broken contracts. (2) The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded. (3) The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely. A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility. The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.