Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems

2025-11-03

AI Summary

Existing retrieval methods rely solely on coarse-grained, agent-level descriptions for routing, neglecting fine-grained tool functionalities—leading to low precision in tool-to-agent matching within multi-agent systems. To address this, we propose Tool-to-Agent Retrieval (TAR), a framework that constructs a unified vector space by jointly embedding tools and their associated agents, while explicitly modeling metadata relationships to prevent context dilution. TAR enables collaborative, fine-grained retrieval at both the tool and agent levels. Through joint representation learning across multiple models and relation-enhanced vector space modeling, TAR achieves substantial performance gains on LiveMCPBench: Recall@5 improves by 19.4% and nDCG@5 by 17.7%. This work establishes a novel paradigm for precise tool routing in large-scale LLM-based multi-agent systems.

Technology Category

• Multiagent Systems: Modeling other Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances in LLM Multi-Agent Systems enable scalable orchestration of sub-agents, each coordinating hundreds or thousands of tools or Model Context Protocol (MCP) servers. However, existing retrieval methods typically match queries against coarse agent-level descriptions before routing, which obscures fine-grained tool functionality and often results in suboptimal agent selection. We introduce Tool-to-Agent Retrieval, a unified framework that embeds both tools and their parent agents in a shared vector space and connects them through metadata relationships. By explicitly representing tool capabilities and traversing metadata to the agent level, Tool-to-Agent Retrieval enables granular tool-level or agent-level retrieval, ensuring that agents and their underlying tools or MCP servers are equally represented without the context dilution that arises from chunking many tools together. Evaluating Tool-to-Agent Retrieval across eight embedding models, our approach achieves consistent improvements of 19.4% in Recall@5 and 17.7% in nDCG@5 over previous state-of-the-art agent retrievers on the LiveMCPBench benchmark.

Problem

• Improves agent selection by embedding tools in shared vector space
• Addresses suboptimal routing from coarse agent-level descriptions
• Enables granular retrieval of tools or agents through metadata relationships

Innovation

• Embeds tools and agents in shared vector space
• Connects tools to agents via metadata relationships
• Enables granular retrieval at tool or agent level