Visual-to-Haptic Augmentation in XR: A Wearable Glove for Perceptual Grounding in Multimodal Interaction

2026-08-10

AI Summary

Current XR systems underutilize haptic perception and lack effective visuo-haptic multimodal interaction mechanisms. This work proposes a wearable haptic glove paired with a dedicated visuo-haptic mapping algorithm that translates spatiotemporal visual information from images and videos into distributed vibration patterns by fusing motion, edge, and luminance features. The system generates dynamic intensity maps aligned with 29 actuators to deliver event-driven tactile feedback. Built upon a modular four-layer architecture—encompassing the XR environment, media processing, feature transformation, and embedded hardware—user studies demonstrate that the approach significantly enhances realism, immersion, and visuo-haptic consistency in dynamic scenes, exhibiting particularly robust and pronounced performance during dynamic visual events.

Abstract

Extended Reality (XR) systems increasingly deliver high-fidelity visual and auditory experiences, yet tactile perception remains comparatively underutilized as a modality for enriching embodied interaction. This work presents a visual-to-haptic wearable glove and a feature-based visual-to-haptic mapping algorithm that translates spatial and temporal visual features from images and videos into distributed vibrotactile patterns. The proposed method extracts motion, edge, and brightness cues and fuses them into actuator-level intensity maps aligned with a 29-actuator glove arranged in a five-by-seven layout. The system is implemented through a modular four-layer architecture comprising the XR environment, media content handling, visual-to-haptic processing, and embedded haptic hardware. A within-subject user study (N = 20) compared visual-only interaction with visual-plus-haptic augmentation across texture-based and dynamic video scenarios. Results indicate that tactile augmentation significantly improves perceived realism in dynamic video scenarios and enhances immersion and visual-tactile correspondence across conditions, with stronger and more consistent effects observed for dynamic visual events. While the current implementation operates in a single-user, offline-synchronized configuration, the findings demonstrate that vision-driven tactile augmentation can function as a perceptual enhancement layer within multimodal XR systems. Such a layer may provide a foundation for future socially enriched XR environments where coherent multisensory grounding supports higher-level interaction and communication.

Problem

• tactile perception
• multimodal interaction
• extended reality
• haptic augmentation
• perceptual grounding

Innovation

• visual-to-haptic mapping
• wearable haptic glove
• multimodal XR
• vibrotactile feedback
• perceptual grounding