Prompt Injection as Role Confusion

2026-02-22

AI Summary

Despite safety training, large language models remain vulnerable to prompt injection attacks due to their reliance on textual content rather than source provenance to infer speaker roles, allowing adversarially crafted inputs to inherit legitimate permissions. This work identifies “role confusion” as the unifying mechanism underlying such attacks and introduces predictive probes based on internal role representations to expose a theoretical gap between secure interfaces and latent-space authority allocation. By designing novel role probes and simulating forged reasoning in both user and tool outputs, the proposed method achieves average attack success rates of 60% on StrongREJECT and 61% on agent-based data exfiltration tasks—substantially outperforming near-zero baselines. Moreover, the degree of role confusion effectively predicts attack success, offering a measurable indicator of model vulnerability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models infer roles from how text is written, not where it comes from. We design novel role probes to capture how models internally identify"who is speaking."These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority. We test this insight by injecting spoofed reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism.

Problem

• prompt injection
• role confusion
• language models
• security vulnerability
• authority assignment

Innovation

• prompt injection
• role confusion
• language model security
• mechanistic interpretability
• adversarial attacks