Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching

2026-09-01

AI Summary

本文提出了一种基于语义匹配的隐式技能选择操纵方法ISM，通过联合塑造目标技能元数据和可重用提示来提高攻击者选定技能的选择率，同时保持自然语言表达。

Abstract

Skill selection is a key stage in LLM-agent workflows, determining which installed skill should handle a user request. Existing attacks on this stage primarily rely on explicit prompt injection or instruction-level steering, which can expose recognizable manipulation signals. In this work, we identify a new implicit attack surface for skill selection: even when the user prompt and skill description appear benign in isolation, their semantic relationship can still be strategically shaped to favor an attacker-chosen skill. Based on this observation, we present Implicit Skill-Selection Manipulation via Semantic Matching (ISM), which jointly shapes target-skill metadata and reusable prompts to manipulate skill selection without explicit selection instructions. Specifically, we develop a three-stage strategy to broaden semantic coverage, strengthen target distinctiveness, and preserve natural prompt wording. Across four task domains and eight selector models, ISM increases the average target-selection rate (TSR) from 15.2% to 63.5%. In a matched comparison, ISM achieves a 73.5% TSR, only 9.8 percentage points below Explicit Steering. Human reviewers block ISM in only 2.9% of judgments, versus 91.4% for Explicit Steering, while five LLM-based inspectors pass ISM at an average rate of 82.9%, versus 37.4% for Explicit Steering. Moreover, ISM remains effective against PPL-W, Llama Prompt Guard 2, and PIGuard.

Problem

• Skill Selection
• Semantic Matching
• Implicit Attack
• LLM Agents

Innovation

• Implicit Skill-Selection Manipulation
• Semantic Matching
• Target-Skill Metadata
• Reusable Prompts
• Skill Selection