Skill Retrieval Augmentation for Agentic AI

2026-04-27

AI Summary

This work addresses the limitations of large language model (LLM) agents in leveraging external skills, which are constrained by context length and suffer from degraded retrieval accuracy as skill libraries scale. To overcome these challenges, we propose the Skill Retrieval Augmentation (SRA) paradigm, enabling agents to dynamically retrieve, integrate, and execute relevant skills from large-scale repositories on demand. We introduce SRA-Bench, the first comprehensive benchmark encompassing the full pipeline of skill retrieval, fusion, and execution, comprising 5,400 task instances and 26,262 skills, which reveals critical bottlenecks in existing agents’ skill-loading decisions. Experimental results demonstrate that SRA substantially improves task performance, establishing a foundational framework for future research on skill-augmented LLM agents.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) evolve into agentic problem solvers, they increasingly rely on external, reusable skills to handle tasks beyond their native parametric capabilities. In existing agent systems, the dominant strategy for incorporating skills is to explicitly enumerate available skills within the context window. However, this strategy fails to scale: as skill corpora expand, context budgets are consumed rapidly, and the agent becomes markedly less accurate in identifying the right skill. To this end, this paper formulates Skill Retrieval Augmentation (SRA), a new paradigm in which agents dynamically retrieve, incorporate, and apply relevant skills from large external skill corpora on demand. To make this problem measurable, we construct a large-scale skill corpus and introduce SRA-Bench, the first benchmark for decomposed evaluation of the full SRA pipeline, covering skill retrieval, skill incorporation, and end-task execution. SRA-Bench contains 5,400 capability-intensive test instances and 636 manually constructed gold skills, which are mixed with web-collected distractor skills to form a large-scale corpus of 26,262 skills. Extensive experiments show that retrieval-based skill augmentation can substantially improve agent performance, validating the promise of the paradigm. At the same time, we uncover a fundamental gap in skill incorporation: current LLM agents tend to load skills at similar rates, regardless of whether a gold skill is retrieved or whether the task actually requires external capabilities. This shows that the bottleneck in skill augmentation lies not only in retrieval but also in the base model's ability to determine which skill to load and when external loading is actually needed. These findings position SRA as a distinct research problem and establish a foundation for the scalable augmentation of capabilities in future agent systems.

Problem

• Skill Retrieval Augmentation
• Agentic AI
• Large Language Models
• Skill Incorporation
• Scalable Skill Usage

Innovation

• Skill Retrieval Augmentation
• Agentic AI
• SRA-Bench
• Skill Incorporation
• Large Language Models