ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior

2025-12-05

AI Summary

Multimodal large language models (MLLMs) are vulnerable to indirect prompt injection (IPI) attacks, where malicious instructions are implicitly embedded in images, videos, or audio—posing challenges for existing text-centric defenses due to poor transferability and generalization. This paper introduces the first representation-space regulation framework tailored for MLLMs: we identify that instruction-following behavior concentrates in a specific latent subspace, enabling disentanglement of safety constraints from performance-degrading directions. Our method integrates adaptive-strength activation steering, on-demand activation mechanisms, and post-hoc filtering verification to enable precise, minimal intervention in the representation space. It further unifies steering-based activation modulation, lightweight IPI detection, and multimodal post-processing. Crucially, it preserves original model capabilities while significantly enhancing robustness and cross-modal generalization against diverse IPI attacks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal Large Language Models (MLLMs) are increasingly vulnerable to multimodal Indirect Prompt Injection (IPI) attacks, which embed malicious instructions in images, videos, or audio to hijack model behavior. Existing defenses, designed primarily for text-only LLMs, are unsuitable for countering these multimodal threats, as they are easily bypassed, modality-dependent, or generalize poorly. Inspired by activation steering researches, we hypothesize that a robust, general defense independent of modality can be achieved by steering the model's behavior in the representation space. Through extensive experiments, we discover that the instruction-following behavior of MLLMs is encoded in a subspace. Steering along directions within this subspace can enforce adherence to user instructions, forming the basis of a defense. However, we also found that a naive defense direction could be coupled with a utility-degrading direction, and excessive intervention strength harms model performance. To address this, we propose ARGUS, which searches for an optimal defense direction within the safety subspace that decouples from the utility degradation direction, further combining adaptive strength steering to achieve a better safety-utility trade-off. ARGUS also introduces lightweight injection detection stage to activate the defense on-demand, and a post-filtering stage to verify defense success. Experimental results show that ARGUS can achieve robust defense against multimodal IPI while maximally preserving the MLLM's utility.

Problem

• Defends MLLMs against multimodal indirect prompt injection attacks.
• Searches optimal defense direction to decouple safety from utility degradation.
• Uses adaptive steering and detection for robust safety-utility trade-off.

Innovation

• Steering instruction-following behavior in representation space
• Searching optimal defense direction decoupled from utility degradation
• Combining adaptive strength steering with on-demand activation