ToolGen: Unified Tool Retrieval and Calling via Generation

2024-10-04

arXiv.org

AI Summary

Existing context-retrieval-based approaches for enabling large language models (LLMs) to autonomously invoke external tools are constrained by fixed context length and suffer from poor scalability. This paper proposes ToolToken: a method that encodes each tool as a learnable, unique token, allowing the LLM to directly generate tool invocations—including arguments—via standard autoregressive decoding. For the first time, this internalizes tool knowledge into the model’s parameters, eliminating explicit retrieval entirely and enabling zero-overhead integration of arbitrarily large tool sets. We evaluate on a large-scale benchmark comprising over 47,000 tools, demonstrating significant improvements over state-of-the-art methods in both tool retrieval accuracy and task completion rate. Moreover, ToolToken natively supports chain-of-thought reasoning and joint optimization with reinforcement learning, facilitating more robust and adaptive tool use.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains. By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs.

Problem

• Enabling LLMs to autonomously interact with external tools
• Overcoming context length limits for tool descriptions
• Eliminating separate inefficient tool retrieval mechanisms

Innovation

• Integrates tool knowledge into LLM parameters
• Generates tool calls via token prediction
• Eliminates separate tool retrieval steps