Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence

2026-05-01

AI Summary

This work addresses the lack of haptic feedback in mixed reality, which causes visuo-haptic inconsistency and diminishes immersion. The authors propose a generative artificial intelligence approach that dynamically transforms everyday physical objects into geometrically compatible passive haptic props based on user-provided text prompts, while simultaneously generating virtual content aligned with the props’ shapes. This method represents the first framework to integrate textual semantics with geometric constraints for shape-aware virtual content generation, enabling dynamic haptic interaction and creative expression. Experimental results demonstrate that the system achieves high shape similarity and prompt fidelity, significantly enhancing users’ perceived realism, sense of presence, and interactive engagement.

Abstract

Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a MR system based on generative artificial intelligence (AI) that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our AI pipeline performs generation and anchoring of virtual assets that align with the shape of physical props, allowing us to study how virtual content generation behaves under geometric and prompt-based constraints. We evaluate Prop-Chromeleon's effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user study further showcases Prop-Chromeleon's improvements in perceived realism, immersion, and enjoyment compared to static baselines. These results show that shape-aware generation can support both believable haptic interaction and creative engagement in MR.

Problem

• Mixed Reality
• haptic feedback
• visual-tactile consistency
• generative AI
• passive haptic props

Innovation

• generative AI
• adaptive haptic props
• mixed reality
• shape-aware generation
• text-to-3D