SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

2026-05-07

AI Summary

This work addresses the challenges of efficiency and accuracy in large-scale skill retrieval for LLM-based agents. We propose SkillRet—the first large-scale benchmark for skill retrieval—comprising 17,810 semantically annotated structured skills and over 60,000 query samples, organized under a two-level taxonomy with disjoint training and evaluation skill pools. Leveraging this benchmark, we develop an optimized retrieval model that integrates hierarchical semantic modeling, diverse retrieval architectures, and task-oriented fine-tuning. Our approach achieves a 13.1-point improvement in NDCG@10 over the strongest baseline and outperforms off-the-shelf models by 16.9 points, demonstrating the effectiveness and necessity of SkillRet for advancing skill retrieval in agent systems.

Abstract

As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 17,810 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,997 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved: off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet substantially improves performance, improving NDCG@10 by +13.1 points over the strongest prior retriever and by +16.9 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems.

Problem

• skill retrieval
• LLM agents
• large-scale benchmark
• retrieval challenge
• agent skills

Innovation

• skill retrieval
• LLM agents
• large-scale benchmark
• structured taxonomy
• retrieval fine-tuning