Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

2026-07-08

AI Summary

This work addresses the high latency, low reliability, and inconsistent outputs observed in production LLM agents due to repetitive generation of identical procedural code. It proposes the first industrial-scale framework for tool fabrication and self-evolution in LLM systems: during deployment, frequently used standard operations are compiled into verified, version-controlled callable tools, which are prioritized at runtime over code generation, with the latter reserved only for cases where no suitable tool exists. By replacing redundant reasoning with structured tool invocation, this approach substantially enhances system stability and auditability while effectively surfacing data drift. Evaluated on a real-world alert triage system, tool calling reduces p50 latency by 42% and end-to-end error rates by up to 53%; when combined with architectural simplification, latency is further reduced by 62%.

Abstract

Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment. The tool-maker grounds synthesis in the live environment as it collects execution traces, observes backend schemas and values, generates candidate tools, and repairs them against labeled cases. At runtime, the production agent calls these tools directly and falls back to code generation only when needed. We deploy the approach in a Fulfillment Center alarm-triage system, where an agent diagnoses alarms against a 44-node SOP over heterogeneous metric backends. In production, tool calls reduce p50 latency by 42%. On 1,500 historical alarms, they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, they also enable a simpler direct-call architecture, reducing p50 latency by a further 62% in a controlled ablation. Versioned tools also improve auditability and expose specification gaps and upstream data drift. Our results show that self-evolving agents can make industrial LLM systems faster, more reliable, and easier to operate.

Problem

• LLM agents
• low-latency systems
• code regeneration
• reliability
• latency

Innovation

• tool-making
• self-evolving agents
• low-latency LLM systems
• structured tool calls
• versioned tools