ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations

2025-01-28

AI Summary

Non-structured, inconsistent, and incomplete REST API documentation impedes automated tool construction by AI systems, necessitating costly manual intervention. Method: We propose the first automated framework for generating tools from API documentation, featuring (1) an LLM-driven pipeline for multi-format document understanding and structured extraction; (2) a verification-augmented knowledge base enabling inference-based completion of missing parameters and response fields; and (3) the first open-source API extraction benchmark—comprising 167 documents and 744 endpoints—with fine-grained JSON Schema annotations. Results: Our method achieves significant improvements over baselines on the API Extraction Benchmark. It has been successfully deployed in a domain-specific AI agent for glycobiology research. The open-source toolchain enables zero-shot, rapid integration of scientific REST APIs into AI workflows.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

LLM-based tool agents offer natural language interfaces, enabling users to seamlessly interact with computing services. While REST APIs are valuable resources for building such agents, they must first be transformed into AI-compatible tools. Automatically generating AI-compatible tools from REST API documents can greatly streamline tool agent development and minimize user learning curves. However, API documentation often suffers from a lack of standardization, inconsistent schemas, and incomplete information. To address these issues, we developed extbf{ToolFactory}, an open-source pipeline for automating tool generation from unstructured API documents. To enhance the reliability of the developed tools, we implemented an evaluation method to diagnose errors. Furthermore, we built a knowledge base of verified tools, which we leveraged to infer missing information from poorly documented APIs. We developed the API Extraction Benchmark, comprising 167 API documents and 744 endpoints in various formats, and designed a JSON schema to annotate them. This annotated dataset was utilized to train and validate ToolFactory. The experimental results highlight the effectiveness of ToolFactory. We also demonstrated ToolFactory by creating a domain-specific AI agent for glycomaterials research. ToolFactory exhibits significant potential for facilitating the seamless integration of scientific REST APIs into AI workflows.

Problem

• REST API Documentation
• Automated Generation
• Artificial Intelligence Integration

Innovation

• REST API Documentation
• AI Tool Generation
• Error Detection Method