SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces

2026-05-12

AI Summary

This work addresses a critical gap in existing safety evaluations by identifying a novel attack surface introduced through reusable skills: even when user requests are benign, skill materials or local artifacts can inadvertently induce agents to perform unsafe actions. The study presents SkillSafetyBench, the first systematic benchmark encompassing six risk domains, 30 safety categories, and 47 tasks, designed to assess the safety of large language model agents during skill invocation. Through adversarial case design, rule-based validators, and comparative experiments across multi-agent setups and model backends, the research demonstrates that localized, non-user-originated attacks can reliably trigger unsafe behaviors. Moreover, failure modes exhibit significant variation across domains, attack strategies, and agent architectures, revealing that safety depends not only on model alignment but also on skill parsing, contextual trust assumptions, and execution environments.

Abstract

Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, task-relevant skill materials or local artifacts can steer an agent toward unsafe actions. We present SkillSafetyBench, a runnable benchmark for evaluating such skill-mediated safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that localized non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments.

Problem

• agent safety
• skill-facing attacks
• adversarial evaluation
• modular skills
• unsafe behavior

Innovation

• skill-mediated safety
• adversarial benchmarking
• agent attack surfaces
• executable environments
• safety evaluation