SkillsMetric: Mapping the Detection Boundary of Static Analysis for Malicious Agent Skills

2026-08-09

AI Summary

This study addresses the lack of effective methods for evaluating the security of skill packages employed by large language model (LLM) agents, as existing static analysis approaches struggle to comprehensively identify potential malicious behaviors. To bridge this gap, the work proposes a five-dimensional static analysis framework that holistically scores skill packages based on pattern density, statistical anomalies, data-flow taint propagation, import irregularities, and semantic mismatches between declared capabilities and actual implementation. The authors also introduce SkillMD-138K, the first adversarial evaluation dataset specifically designed for agent skills. Experimental results demonstrate strong overall detection performance with an AUC of 0.93 and an F1 score of 73.4%, achieving a 93% detection rate for data exfiltration and steganographic payloads. However, the study reveals fundamental blind spots in static analysis regarding host compromise and prompt injection attacks, offering critical insights for developing layered defense strategies.

Abstract

Agent Skills---structured packages of instructions and scripts that augment LLM-based agents---are rapidly proliferating, yet their security properties remain under-explored. We present \textsc{SkillsMetric}, a five-stage static analysis framework that scores skill packages along pattern density, statistical anomaly, dataflow taint, import anomaly, and capability mismatch dimensions. We construct an adversarial evaluation dataset of 2{,}266 skills spanning 16~attack types across code-level, system-level, and semantic-level threats, and evaluate on the full SkillMD-138K corpus. Our framework achieves an AUC of 0.93 and 5-fold cross-validated F1 of 73.4\%$\pm$0.5\%, with strong detection of data exfiltration (93\%) and steganographic payloads (93\%). Crucially, we identify fundamental blind spots: \emph{host destruction} attacks using common shell commands evade all five stages (0\% detection), and \emph{prompt injection} via natural-language manipulation achieves only 42\% detection. These findings establish that static analysis alone is insufficient for skill security, motivating defense-in-depth architectures that combine fast static pre-screening with semantic review.

Problem

• Agent Skills
• static analysis
• malicious detection
• security blind spots
• adversarial evaluation

Innovation

• static analysis
• agent skills
• adversarial evaluation
• dataflow taint
• defense-in-depth