SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

2026-06-14

AI Summary

This work addresses the critical gap in security evaluation of open-source large language model (LLM) agents, which currently lack effective mechanisms to assess semantic-level instruction threats and multi-agent collaboration risks—vulnerabilities often undetectable by conventional code-scanning tools. To this end, we propose SARS (Skill Agent Risk Scoring), a novel framework that integrates CVSS v4.0 vector decomposition with an LLM-as-Judge architecture to establish a five-dimensional risk metric. SARS introduces ClawHub, a dual-view comparative mechanism that concurrently displays LLM-based assessments alongside official marketplace judgments. Evaluated on 78 malicious and 22 benign skills, our approach achieves zero false negatives and zero false positives, respectively, outperforming the state-of-the-art SKILLSIEVE by a 15% detection rate improvement and attaining 100% detection accuracy against prompt injection and memory poisoning attacks. The framework is publicly deployed with a leaderboard on Hugging Face.

Abstract

Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted. The gap we fill: existing scanners operate at the code layer and are structurally blind to instruction-layer and multi-agent risk - natural-language directives that hijack an agent, exfiltrate data through encoded side channels, or chain harm across pipelines - so what is needed is a semantic, multi-dimensional vetting system rather than another signature matcher. We present SKILLVETBENCH, a live public leaderboard on Hugging Face that uses an LLM-as-Judge to vet agent skills. What is new: SARS (Skill Agentic Risk Score), a five-dimensional agentic-risk metric with a principled weighted formula for instruction-following systems. What is integrated: full CVSS v4.0 vector decomposition and a ClawHub dual-view that places our LLM-generated review beside the official marketplace verdict. What is demonstrated: drawing on our companion benchmark paper [ 1], the LLM-as-Judge stage achieves zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls, while the best static baseline (SKILLSIEVE) still misses 15%; for instruction-layer categories such as Prompt Injection and Memory Poisoning, conventional tools miss between 89% and 100% of threats (e.g., CODEBERT detects none of nine memory-poisoning skills). Detection rates vary from 35% to 95% across four LLM evaluators, motivating ensemble scoring in production deployments.

Problem

• LLM agent skills
• security risk evaluation
• instruction-layer threats
• multi-agent risk
• open-source LLM

Innovation

• LLM-as-Judge
• Skill Agentic Risk Score
• instruction-layer security
• multi-agent risk
• semantic vetting