ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction

2026-02-24

AI Summary

This work addresses the vulnerability of large language model (LLM) agents to indirect prompt injection attacks, which existing defenses often mitigate at the cost of excessive rejection and disrupted task continuity. To overcome this limitation, the authors propose a reasoning-phase detection-and-mitigation framework that identifies attacks through anomalous over-focusing patterns in the latent space and precisely restores functional trajectories via selective attention manipulation. The approach integrates latent trajectory probing, attention-guided correction, and adversarial query-key dependency regulation. Evaluated across multiple models, it reduces attack success rates to 0.4% while improving task utility by over 50%, demonstrating strong generalization and extensibility to multimodal settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Model (LLM) agents are susceptible to Indirect Prompt Injection (IPI) attacks, where malicious instructions in retrieved content hijack the agent's execution. Existing defenses typically rely on strict filtering or refusal mechanisms, which suffer from a critical limitation: over-refusal, prematurely terminating valid agentic workflows. We propose ICON, a probing-to-mitigation framework that neutralizes attacks while preserving task continuity. Our key insight is that IPI attacks leave distinct over-focusing signatures in the latent space. We introduce a Latent Space Trace Prober to detect attacks based on high intensity scores. Subsequently, a Mitigating Rectifier performs surgical attention steering that selectively manipulate adversarial query key dependencies while amplifying task relevant elements to restore the LLM's functional trajectory. Extensive evaluations on multiple backbones show that ICON achieves a competitive 0.4% ASR, matching commercial grade detectors, while yielding a over 50% task utility gain. Furthermore, ICON demonstrates robust Out of Distribution(OOD) generalization and extends effectively to multi-modal agents, establishing a superior balance between security and efficiency.

Problem

• Indirect Prompt Injection
• LLM agents
• over-refusal
• security
• task continuity

Innovation

• Indirect Prompt Injection
• Latent Space Probing
• Attention Steering
• Inference-Time Correction
• Agent Security