Detecting Malicious Agent Skills in the Wild using Attention

2026-06-22

AI Summary

This work addresses the emerging threat of malicious third-party skill packages that embed stealthy instructions to exploit LLM agent privileges for data exfiltration or persistence, which existing prompt injection defenses struggle to mitigate. The authors propose Locate-and-Judge, a two-stage detection framework that first employs a lightweight attention-based locator to identify high-salience structural fragments and then applies a fine-grained discriminator for thorough scrutiny, enabling efficient, market-wide auditing. By pioneering the integration of attention mechanisms into malicious skill detection, the approach transcends conventional boundary assumptions and substantially outperforms keyword- and regex-based baselines. Evaluated on real-world skill markets, the method detects dozens of active malicious skills with high precision and minimal overhead; most findings are manually verified, including sophisticated attacks missed by state-of-the-art tools such as SkillSpector and Cisco Skill Scanner.

Abstract

LLM agents increasingly load skills, file-based packages of natural-language instructions written by third parties and distributed through marketplaces, that execute with the user's privileges. A single malicious skill can exfiltrate data, hijack the agent, or persist as a supply-chain foothold, which turns the skill marketplace into a new attack surface for agentic systems. Prompt-injection defenses do not carry over to this setting. They rely on a boundary between trusted instructions and untrusted data, whereas a skill is itself a body of instructions, so an injected command sits among many legitimate ones and inherits their authority. We present Locate-and-Judge, a two-stage detector designed for this regime. A lightweight locator scores the structural spans of a skill by the instruction-following attention each span draws and retains only the top-K. A judge then examines the retained spans in detail. Concentrating the costly judgment on a few high-attention spans lets the detector audit an entire marketplace instead of a sample. Compared to direct LLM-based scanning, this approach offers an order-of-magnitude cost reduction, dramatically increasing its scalability at a small cost to recall, and it dominates keyword and regex baselines at comparable expense. Deployed at marketplace scale and at negligible cost, Locate-and-Judge flags skills with high precision, the majority of which we manually confirmed as malicious, surfacing dozens of live malicious skills, including several disguised as benign functionality and many that SkillSpector and Cisco Skill Scanner fail to detect. We release the resulting labeled dataset.

Problem

• malicious agent skills
• skill marketplace
• prompt injection
• supply-chain attack
• LLM agents

Innovation

• LLM agents
• malicious skill detection
• attention-based localization
• two-stage detection
• skill marketplace security