Skill Coverage: A Test Adequacy Metric for Agent Skills

2026-06-09

AI Summary

Current evaluation methods struggle to finely assess how large language model agents actually utilize reusable skills and why they fail. This work proposes "skill coverage" as a test adequacy metric tailored to agent skills, translating natural language skill instructions into semi-structured behavioral constraints and evaluating whether these constraints are covered and successfully executed based on execution traces. This approach decouples skill usage from task outcomes, enabling actionable failure attribution. Experiments on SkillsBench reveal that existing agents cover only 38.66%–45.51% of skill constraints; further, reinforcing skills based on failed constraints yields an average task recovery rate of 16.0% across previously failed tasks.

Abstract

Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts. Existing evaluations primarily assess skills through task level outcomes, yet task success alone does not reveal which parts of a skill have been exercised or which remain untested. We introduce skill coverage, a test adequacy metric that treats the skill artifact as the object under test. Our approach extracts observable skill behavior constraints from skill documents and measures whether an agent trajectory provides sufficient evidence to exercise and evaluate each constraint. Skill coverage uses a binary cover or not cover judgment, which reports whether a documented behavior has been exercised with sufficient observable evidence, without assigning an additional outcome label to the behavior. Applying skill coverage to SkillsBench reveals that existing benchmark executions cover only 39.90 to 43.98% of skill behavior constraints, suggesting that current benchmark tasks leave large portions of documented skill guidance untested. These findings show that successful task completion does not imply adequate testing of the skill artifact, highlighting skill coverage as a measure of how thoroughly agent skills are tested.

Problem

• skill coverage
• test adequacy
• agent skills
• behavior constraints
• LLM agents

Innovation

• skill coverage
• test adequacy
• behavior constraints
• LLM agents
• trajectory analysis