🤖 AI Summary
While existing OpenAPI specifications are structurally compliant, they often fail to meet the semantic requirements of AI agents for effective task planning and API invocation, leading to systematic failures. This work proposes Hermes, a multi-agent large language model system that pioneers the application of multi-agent LLMs to endpoint-level semantic readiness assessment of large-scale, production-grade APIs. Hermes analyzes OpenAPI specifications, identifies RESTful and documentation-related code smells, and generates interpretable diagnostic reports. Evaluated on 16 real-world APIs encompassing approximately 600 endpoints, Hermes detected 2,450 such smells, with high precision confirmed by developer validation. The findings have prompted enterprises to incorporate semantic documentation quality into their API governance frameworks and to redefine API design standards for AI agent compatibility.
📝 Abstract
The growing adoption of AI agents and the Model Context Protocol (MCP) has motivated organizations to expose existing REST APIs as agent-consumable tools. In our industrial context, this initiative targeted an ecosystem of 16 production APIs comprising approximately 600 endpoints. Although these APIs were stable and widely used within a microservice architecture, early proof-of-concept experiments revealed systematic failures in task planning, tool selection, and payload construction when accessed through MCP-based agents. Rather than attributing these failures to model limitations alone, we conducted an ecosystem-scale empirical assessment of the underlying OpenAPI documentation. We developed Hermes, a multi-agent LLM-based system that detects documentation and REST-related smells at the endpoint level and generates explainable diagnostic reports. The large-scale evaluation identified 2,450 smells across 600 endpoints, with deficiencies present in all analyzed operations. Practitioner validation confirmed high agreement with the detected issues while also revealing contextual trade-offs in remediation decisions. The findings suggested that structural validity within microservice environments does not guarantee semantic readiness for agent-based consumption. Based on this evidence, the organization revised its adoption strategy, prioritizing selective endpoint adaptation, redefining documentation standards, and integrating automated documentation assessment into API governance workflows. This case illustrates how systematic artifact-level evaluation can function as a strategic decision-support mechanism, reducing technological risk and guiding evidence-based AI adoption in industrial software ecosystems.