Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval

2026-08-17

AI Summary

This study addresses the lack of capability grounding and parameter consistency in pseudo-query generation for agent skill retrieval by proposing the Skill2Query framework. This approach constructs a knowledge graph from skill documentation and employs a three-stage mechanism—style imitation, template generation, and parameter filling—to synthesize high-quality pseudo-queries for index augmentation, query expansion, and retriever fine-tuning. Experiments demonstrate that 700,000 generated pseudo-queries improve average Recall@1 by 6.70%, significantly outperforming existing baselines while effectively enhancing agent task success rates. Consequently, this work establishes a novel structured augmentation paradigm for skill retrieval, resolving critical alignment issues between natural language queries and executable agent capabilities through systematic knowledge-grounded synthesis.

Abstract

Pseudo-query generation can alleviate the supervision bottleneck for agent skill retrieval, but existing document-level approaches typically leave the rich internal relations among capabilities, parameters, and usage examples implicit. As a result, generated queries may be topically relevant to a skill while lacking capability grounding and parameter consistency, raising the question of whether explicitly exploiting a skill document's internal structure can produce more effective retrieval signals. We therefore propose Skill2Query, a framework that first parses a skill document into a Skill Knowledge Graph and then generates pseudo-queries through a three-stage process including style mimicking, query template generation, and parameter filling. The generated queries can be used for offline index augmentation, online query expansion, and retriever training. Four benchmarks (TheoremQA, LogicBench, ToolQA, and CHAMP) are used to evaluate Skill2Query with large-scale skill candidate pools across multiple downstream applications, including skill retrieval, retriever training, and end-to-end agent execution. Using nearly 30K skills across diverse domains, we generate 700K category-diverse pseudo-queries. Skill2Query consistently improves sparse, dense, and skill-routing retrieval, with an average Recall@1 gain of 6.70 percentage points across retrieval settings. Skill2Query-generated training data also achieves the best Recall@1 and nDCG@1 among the evaluated generation baselines. Further evaluations with multiple LLM backends demonstrate that improved skill retrieval translates into higher agent task success rates. Code and resources are available at https://github.com/MatZaharia/Skill2Query.

Problem

• Agent Skill Retrieval
• Pseudo-query Generation
• Skill Structure
• Supervision Bottleneck

Innovation

• Pseudo-query Generation
• Skill Knowledge Graph
• Agent Skill Retrieval
• Structured Parsing
• Index Augmentation