Devil in the Lens: Analyzing and Defending Physical Prompt Injection Against Vision-Language Models on Wearable Devices

2026-07-11

AI Summary

This study addresses a critical security vulnerability in vision-language models (VLMs) deployed on wearable devices, which are susceptible to physical-world prompt injection attacks triggered by malicious text in the environment, leading to the generation of harmful or fabricated content. The work presents the first systematic characterization and modeling of such physically mediated indirect prompt injection threats, identifying six realistic attack vectors. Through an evaluation involving over 200 real-world images captured by AI-powered smart glasses, the authors assess the susceptibility of twelve state-of-the-art VLMs, demonstrating attack success rates of 96% in simulated settings and 60% in real-world conditions. To mitigate this risk, the paper proposes a dual-layer defense framework that integrates external textual filtering with internal semantic anomaly detection, substantially enhancing model robustness and safety against such adversarial exploits.

Abstract

Vision-Language Models (VLMs) are rapidly deployed on human-facing wearable devices such as smart glasses to enable multimodal perception and AI-assisted decision-making. While prior research has demonstrated the risks of visual prompt injection into digital image inputs of VLMs, the unique security challenges posed by the increasing integration between physical environments and wearable intelligence, such as those embodied in VLM-enabled AI glasses, remain underexplored. Toward understanding and modeling such threats, our work characterizes how malicious textual information embedded in physical environments introduces a high-priority visual channel for indirect prompt injection, where scene texts that hinder or evade human perception could hijack VLM models' behavior. Such \textit{Physical Prompt Injection Attacks} can not only disrupt normal tasks of VLM-enabled wearable devices, but also steer models to produce profane, biased, or even untruthful outputs. Using physically captured photos from AI glasses in over 200 real-world environments, our analysis identifies 6 representative threat vectors of physically injected prompts, and further evaluates their impacts on 12 VLM models. Results show that these attacks consistently manipulate model outputs across integrity- and safety-critical tasks, achieving attack success rates of up to 96\% and 60\% in simulated and real-world settings. Our analysis confirms that multiple models exhibit excessive blind trust in environmental text, ignoring the actual visual context and producing completely opposite summaries or directives. We further propose two targeted defense strategies, including a mask-based external filter and a semantic-vector-based internal detector, to effectively reduce the success rate and safety impact of these attacks.

Problem

• Physical Prompt Injection
• Vision-Language Models
• Wearable Devices
• Adversarial Attacks
• Multimodal Security

Innovation

• Physical Prompt Injection
• Vision-Language Models
• Wearable AI Security
• Adversarial Text in Physical World
• Multimodal Defense