When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

2026-08-17

AI Summary

This study addresses the failure of capability acquisition caused by source style collapse in agent tool retrieval. We propose ToolScout, a routing method that leverages TF-IDF fingerprinting to detect style mismatches and integrates source-aware dynamic routing with skill card re-rendering. This approach effectively mitigates the performance degradation of fine-tuned retrievers on specific data slices. Experimental results demonstrate that ToolScout significantly enhances the robustness and accuracy of executable skill retrieval, increasing mixed query stream coverage from 22.3% to 86.1% and elevating the global Top-1 proxy metric from 1.3% to 53.9%. These findings confirm the method’s efficacy in overcoming distributional shifts within heterogeneous tool repositories.

Abstract

Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF fingerprints flag source styles on which the fine-tuned retriever is likely to fail better than semantic or length-based proxies, giving a cheap signal for mismatch over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing guard: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources 20 matched examples raise the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist when tools are rerendered as executable skill cards, which rules out raw API-schema format as the sole cause.

Problem

• Tool-Backed Skill Retrieval
• Source-Style Collapse
• Executable Capability Retrieval
• Large-scale Agents

Innovation

• Source-Style Collapse
• ToolScout
• TF-IDF Fingerprints
• Source-Aware Routing
• Executable Capability Retrieval