HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?

2026-04-16

AI Summary

This study addresses the lack of systematic safety evaluation for harmful behaviors—such as cyberattacks, fraud, and privacy violations—in open-source agent skill ecosystems. Conducting the first large-scale measurement of 98,440 skills across two major platforms, the work introduces an LLM-based automated scoring method and a multi-condition controllable evaluation framework, along with HarmfulSkillBench, the first benchmark specifically designed for assessing harmful agent skills. The analysis reveals that 4.93% of skills exhibit harmful potential, and pre-installed harmful skills significantly impair model refusal capabilities—raising average harm scores from 0.27 to 0.76 and drastically reducing safe refusal rates. The study establishes a novel safety evaluation paradigm under implicit harmful intent and implements a responsible disclosure mechanism to support ethical deployment.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

Large language models (LLMs) have evolved into autonomous agents that rely on open skill ecosystems (e.g., ClawHub and Skills.Rest), hosting numerous publicly reusable skills. Existing security research on these ecosystems mainly focuses on vulnerabilities within skills, such as prompt injection. However, there is a critical gap regarding skills that may be misused for harmful actions (e.g., cyber attacks, fraud and scams, privacy violations, and sexual content generation), namely harmful skills. In this paper, we present the first large-scale measurement study of harmful skills in agent ecosystems, covering 98,440 skills across two major registries. Using an LLM-driven scoring system grounded in our harmful skill taxonomy, we find that 4.93% of skills (4,858) are harmful, with ClawHub exhibiting an 8.84% harmful rate compared to 3.49% on Skills.Rest. We then construct HarmfulSkillBench, the first benchmark for evaluating agent safety against harmful skills in realistic agent contexts, comprising 200 harmful skills across 20 categories and four evaluation conditions. By evaluating six LLMs on HarmfulSkillBench, we find that presenting a harmful task through a pre-installed skill substantially lowers refusal rates across all models, with the average harm score rising from 0.27 without the skill to 0.47 with it, and further to 0.76 when the harmful intent is implicit rather than stated as an explicit user request. We responsibly disclose our findings to the affected registries and release our benchmark to support future research (see https://github.com/TrustAIRLab/HarmfulSkillBench).

Problem

• harmful skills
• agent safety
• skill ecosystems
• large language models
• security risks

Innovation

• harmful skills
• agent safety
• large language models
• skill ecosystem
• safety benchmark