Demystifying Agent Skills: Why They Work-Until They Don't

2026-08-14

AI Summary

This study addresses the unclear activation mechanisms and failure boundaries of skills in LLM agents by employing controlled experiments and trajectory analysis to construct a taxonomy comprising three categories and twelve patterns. The research identifies "procedural anchoring" as the dominant mechanism for stable execution, accounting for 65.7% of cases, and quantifies the degradation of retrieval precision as pool size increases. Results demonstrate that skills outperform workflow memory by 6.06 points. Moving beyond traditional aggregate evaluation paradigms, this work systematically elucidates the underlying mechanisms and failure conditions of skill utilization. Consequently, it provides both theoretical foundations and practical guidance for designing reliable self-evolving agents, offering critical insights into optimizing agent architectures through rigorous mechanistic analysis rather than mere performance benchmarking.

Abstract

Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \emph{\textbf{When do skills help, why do they work, and where do they fail?}} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.

Problem

• LLM Agents
• Agent Skills
• Skill Evaluation
• Procedural Anchoring
• Skill Failure

Innovation

• Procedural Anchoring
• Skill Taxonomy
• Trajectory Analysis
• Retrieval Bottleneck
• Agent Skills