SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

2026-06-21

AI Summary

This work addresses the limitations of existing evaluations for large language model (LLM) agents, which predominantly rely on fixed task sets and fail to capture the true value and risks of agent capabilities. The authors propose SkillAudit, a novel skill-centric evaluation framework that decomposes agent skills into skill packages and generates capability-aligned tasks for assessment. Executed within an isolated sandbox environment and judged automatically by LLMs, SkillAudit produces auditable evaluation reports across three dimensions: utility, efficiency/cost, and safety. Its key innovations include a baseline-comparison principle to quantitatively measure utility and cost, and a two-stage safety verification mechanism combining static semantic analysis with dynamic runtime validation. Applied to real-world skill packages across 23 occupational domains, the framework identified security risks in over 7% of skills, demonstrating its effectiveness and necessity.

Abstract

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Problem

• skill evaluation
• large language model agents
• fixed-suite benchmarking
• skill-centered assessment
• agent skills

Innovation

• skill-centered assessment
• automated task generation
• sandboxed evaluation
• baseline comparison principle
• two-stage safety detection