Sound2Hap: Learning Audio-to-Vibrotactile Haptic Generation from Human Ratings

2026-01-18

AI Summary

This work addresses the limitations of existing audio-to-vibration conversion methods, which rely on scenario-specific signal processing rules and fail to generalize across diverse environmental sounds while neglecting perceptual consistency from the user’s perspective. To overcome these challenges, the authors conduct a large-scale user study to construct a dataset of human perceptual ratings on audio–tactile matching. Leveraging this dataset, they propose the first data-driven, low-latency CNN autoencoder trained end-to-end to map arbitrary environmental sounds to coordinated vibrotactile feedback. Experimental results demonstrate that the proposed method significantly outperforms conventional baselines in both audio–vibration alignment and the Haptic Experience Index (HXI), achieving high-quality, perceptually consistent tactile feedback across a wide range of sound categories.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Environmental sounds like footsteps, keyboard typing, or dog barking carry rich information and emotional context, making them valuable for designing haptics in user applications. Existing audio-to-vibration methods, however, rely on signal-processing rules tuned for music or games and often fail to generalize across diverse sounds. To address this, we first investigated user perception of four existing audio-to-haptic algorithms, then created a data-driven model for environmental sounds. In Study 1, 34 participants rated vibrations generated by the four algorithms for 1,000 sounds, revealing no consistent algorithm preferences. Using this dataset, we trained Sound2Hap, a CNN-based autoencoder, to generate perceptually meaningful vibrations from diverse sounds with low latency. In Study 2, 15 participants rated its output higher than signal-processing baselines on both audio-vibration match and Haptic Experience Index (HXI), finding it more harmonious with diverse sounds. This work demonstrates a perceptually validated approach to audio-haptic translation, broadening the reach of sound-driven haptics.

Problem

• audio-to-haptic
• environmental sounds
• vibrotactile feedback
• haptic perception
• sound-driven haptics

Innovation

• audio-to-haptic generation
• perceptual modeling
• data-driven haptics
• CNN autoencoder
• environmental sounds