Feelings, Not Feel: Affective Audio-Visual Pseudo-Haptics in Hand-Tracked XR

2026-02-27

AI Summary

This study addresses the absence of haptic feedback in current controller-free XR hand-tracking interactions, where conventional approaches to simulating realistic tactile sensations have shown limited efficacy. The work proposes reframing pseudo-haptics as an affective feedback channel, systematically modulating users’ emotional states through audiovisual cues—such as motion-modulated surface textures, color glows, and action-coupled sounds—rendered on virtual hands. Using a mixed-reality prototype, the authors conducted a within-subjects experiment comparing unimodal (visual or auditory) and multimodal (audiovisual) conditions with 12 participants. Although no sustained tactile or thermal percepts were elicited, results demonstrated that the “rough–warm” combination significantly reduced valence while increasing arousal, whereas the “smooth–cool” pairing induced a calm and pleasant state, thereby validating the effectiveness of audiovisual pseudo-haptics for affective regulation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Tracking, profiling, and countermeasures against them

Abstract

Hand-tracking enables controller-free XR interaction but does not have the tactile feedback controllers provide. Rather than treating this solely as a missing-sensation problem, we explore whether pseudo-haptic cues on an embodied virtual hand act as tactile or as affect substitutes that shape how interactions feel. We used a mixed reality prototype that keeps the contacted surface visually neutral, rendering cues on the hand with motion modulation for texture, color glow, and movement-coupled sound. In a within-subjects study (n=12), participants experienced 12 conditions (4 effects x 3 modalities: audio, visual, both) and reported subjective affect and cognitive demand. Participants rarely reported sustained tactile, thermal sensations, yet affect shifted systematically: rough-hot lowered valence increasing arousal, while smooth-cold produced calmer pleasant states. These findings suggest that pseudo-haptics in XR may be better understood as an affective feedback channel rather than a direct replacement for physical touch in controller-free systems.

Problem

• pseudo-haptics
• affective feedback
• hand-tracked XR
• tactile feedback
• embodied interaction

Innovation

• pseudo-haptics
• affective feedback
• hand-tracked XR
• multimodal interaction
• embodied virtual hand