PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Models

2026-02-09

AI Summary

This work addresses critical privacy and efficiency challenges in existing extended reality (XR) collaborative systems, which risk exposing sensitive visual data when uploading to cloud-based multimodal large language models (MLLMs) and suffer from inefficient, intrusive environment registration and content synchronization. To overcome these limitations, the study introduces the first XR collaboration framework integrating privacy-aware edge preprocessing with MLLMs, featuring a lightweight, dynamically adaptive spatial registration mechanism and customizable content sharing. By intelligently filtering highly sensitive visual content at the edge, the proposed approach significantly enhances both privacy preservation and collaboration efficiency. Experimental results demonstrate a 90% request satisfaction rate, registration latency under 0.27 seconds, and spatial inconsistency below 3.5 cm. User studies further confirm that over 90% of scenarios achieve automatic sensitive-object filtering without compromising usability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements

Abstract

Multimodal Large Language Models (MLLMs) enhance collaboration in Extended Reality (XR) environments by enabling flexible object and animation creation through the combination of natural language and visual inputs. However, visual data captured by XR headsets includes real-world backgrounds that may contain irrelevant or sensitive user information, such as credit cards left on the table or facial identities of other users. Uploading those frames to cloud-based MLLMs poses serious privacy risks, particularly when such data is processed without explicit user consent. Additionally, existing colocation and synchronization mechanisms in commercial XR APIs rely on time-consuming, privacy-invasive environment scanning and struggle to adapt to the highly dynamic nature of MLLM-integrated XR environments. In this paper, we propose PRISM-XR, a novel framework that facilitates multi-user collaboration in XR by providing privacy-aware MLLM integration. PRISM-XR employs intelligent frame preprocessing on the edge server to filter sensitive data and remove irrelevant context before communicating with cloud generative AI models. Additionally, we introduce a lightweight registration process and a fully customizable content-sharing mechanism to enable efficient, accurate, and privacy-preserving content synchronization among users. Our numerical evaluation results indicate that the proposed platform achieves nearly 90% accuracy in fulfilling user requests and less than 0.27 seconds registration time while maintaining spatial inconsistencies of less than 3.5 cm. Furthermore, we conducted an IRB-approved user study with 28 participants, demonstrating that our system could automatically filter highly sensitive objects in over 90% of scenarios while maintaining strong overall usability.

Problem

• privacy
• Extended Reality (XR)
• Multimodal Large Language Models (MLLMs)
• sensitive data
• collaboration

Innovation

• privacy-aware XR
• multimodal large language models
• edge preprocessing
• lightweight registration
• content synchronization