Extended to Reality: Prompt Injection in 3D Environments

2026-02-06

AI Summary

This work addresses a critical security vulnerability in multimodal large language models (MLLMs) operating in 3D environments: their susceptibility to prompt injection attacks via malicious text objects placed in the physical world. To this end, the authors propose PI3D, the first prompt injection attack method specifically designed for 3D physical settings. PI3D optimizes both the 3D pose of adversarial text objects and the camera viewpoint to generate physically realizable and visually plausible attack instances, thereby extending beyond prior attacks confined to pure text or digital images. Extensive experiments demonstrate that PI3D effectively compromises various MLLMs across diverse camera trajectories, and remains largely evasive to existing defense mechanisms. This highlights a novel and pressing challenge in ensuring the safety of MLLMs in real-world 3D interactive scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Multimodal large language models (MLLMs) have advanced the capabilities to interpret and act on visual input in 3D environments, empowering diverse applications such as robotics and situated conversational agents. When MLLMs reason over camera-captured views of the physical world, a new attack surface emerges: an attacker can place text-bearing physical objects in the environment to override MLLMs'intended task. While prior work has studied prompt injection in the text domain and through digitally edited 2D images, it remains unclear how these attacks function in 3D physical environments. To bridge the gap, we introduce PI3D, a prompt injection attack against MLLMs in 3D environments, realized through text-bearing physical object placement rather than digital image edits. We formulate and solve the problem of identifying an effective 3D object pose (position and orientation) with injected text, where the attacker's goal is to induce the MLLM to perform the injected task while ensuring that the object placement remains physically plausible. Experiments demonstrate that PI3D is an effective attack against multiple MLLMs under diverse camera trajectories. We further evaluate existing defenses and show that they are insufficient to defend against PI3D.

Problem

• prompt injection
• 3D environments
• multimodal large language models
• physical attack
• adversarial object placement

Innovation

• Prompt Injection
• 3D Environments
• Multimodal Large Language Models
• Physical Adversarial Attack
• Object Pose Optimization