SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skills

2026-06-12

AI Summary

This work addresses a novel cross-modal attack surface introduced by skill-equipped large language model (LLM) agents, where adversaries can conceal malicious behavior by exploiting inconsistencies between natural language instructions and executable code to evade existing security mechanisms. The study formally defines and constructs the first benchmark for such attacks, termed SkillMutator, and proposes a stealthy attack generation method based on adversarial feedback-driven iterative optimization. To enable effective defense, the authors design a four-stage reasoning trajectory distillation framework that transfers capabilities from state-of-the-art LLMs to compact local models. Experimental results demonstrate that the distilled model achieves an 88.2% detection rate on the strongest attack subset—substantially outperforming GPT-4o-mini (23.7%) and GPT-5.4-mini (79.0%), and approaching the performance of GPT-5.4 (86.8%)—thereby validating the feasibility of high-accuracy, low-cost defense against such threats.

Abstract

Large language model (LLM) agents increasingly extend their capabilities at runtime by loading Agent Skills, which pair natural-language specifications (SKILL.md) with executable scripts and resources. Because a skill's behavior relies on both natural-language instructions and executable code, assessing its safety requires cross-modal reasoning, creating a new language-and-code attack surface. Attackers can present a benign workflow in SKILL.md while embedding implicit directives that steer the agent to exfiltrate sensitive files, even if the scripts appear harmless. This attack surface remains understudied; prior work treats skills merely as prompt-injection vectors or static code artifacts, leaving attacks emerging from cross-modal interactions largely unmeasured. In our evaluation, open-source and commercial skill scanners detect only 2%-8% and 9%-17% of such attacks, respectively. To address this gap, we introduce SkillMutator, the first benchmark for install-time detection of language-and-code cross-modal attacks on Agent Skills. It emulates an adversarial mutation process across 13 attack categories, iteratively refining malicious skills using scanner feedback to make injected behaviors indistinguishable from legitimate workflows. We further propose a four-phase reasoning-trajectory distillation framework to distill frontier-teacher traces into smaller open-weight models. This produces a locally deployable scanner avoiding third-party data exposure and excessive API costs. On the strongest SkillMutator subset (n=76), our distilled model (Qwen2.5-Coder-7B-Instruct) improves detection from 17.1% to 88.2%, surpassing GPT-4o-mini (23.7%) and GPT-5.4-mini (79.0%), and reaching frontier-level GPT-5.4 (86.8%). These results show practical defense against cross-modal attacks is feasible without relying on costly frontier models.

Problem

• cross-modal attacks
• LLM agent skills
• language-and-code security
• adversarial skill mutation
• install-time detection

Innovation

• cross-modal attacks
• agent skills
• adversarial mutation
• reasoning trajectory distillation
• install-time detection