Demonstrating Visual Information Manipulation Attacks in Augmented Reality: A Hands-On Miniature City-Based Setup

2025-09-02

AI Summary

Visual information manipulation (VIM) attacks in augmented reality (AR) lack effective defensive countermeasures. Method: This work designs and implements an immersive VIM attack demonstration system integrating a physical miniature urban environment with the Meta Quest 3 headset, combining physics-based modeling, spatial mapping, real-time rendering, and computer vision spoofing techniques to construct a reproducible and observable AR security testbed. Results: Empirical evaluation across representative VIM scenarios—including traffic sign tampering and navigation misdirection—demonstrates significant adverse impacts on user decision-making and behavior. To our knowledge, this is the first study to integrate physical miniature environments with consumer-grade AR hardware for VIM attack-defense pedagogy and security assessment. The framework establishes a scalable experimental paradigm and empirical foundation for AR security education, human perception research, and cross-platform defense mechanism development.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Augmented reality (AR) enhances user interaction with the real world but also presents vulnerabilities, particularly through Visual Information Manipulation (VIM) attacks. These attacks alter important real-world visual cues, leading to user confusion and misdirected actions. In this demo, we present a hands-on experience using a miniature city setup, where users interact with manipulated AR content via the Meta Quest 3. The demo highlights the impact of VIM attacks on user decision-making and underscores the need for effective security measures in AR systems. Future work includes a user study and cross-platform testing.

Problem

• Visual Information Manipulation attacks in AR
• Altering real-world visual cues causing confusion
• Underscoring need for AR security measures

Innovation

• Miniature city setup for AR attack simulation
• Meta Quest 3 for manipulated content interaction
• Hands-on demo highlighting VIM attack impacts