A Wearable Multimodal Ultrasound+Inertial System for Real-Time Virtual Reality Interaction

2026-06-16

AI Summary

This work proposes the first fully wearable multimodal virtual reality (VR) interaction system that eliminates reliance on external hardware, addressing the longstanding trade-off between portability and interaction richness. By leveraging only A-mode ultrasound and inertial sensors mounted on the forearm and upper arm, the system employs an end-to-end lightweight framework to enable real-time estimation of hand pose and forearm position, seamlessly integrated into the Unity environment. Built upon the WULPUS platform, it incorporates multimodal learning, real-time communication, and online fine-tuning mechanisms. In offline evaluations, the system achieves 80% ± 6% accuracy in hand pose estimation and 77% ± 7% in forearm position estimation, with over 88% success rate in online interactive tasks. Operating at a mere 19.9 mW, it supports continuous use for more than 2.5 days, demonstrating high accuracy, low power consumption, and robust cross-session stability.

Abstract

A-mode ultrasound (US) is a promising sensing modality for Virtual Reality (VR) interaction, as it enables the mapping of muscular activity into control commands while retaining the benefits of wearable sensing. However, existing approaches still face limitations in terms of wearability and interaction complexity, often relying on external hardware such as cameras. In this work, we propose a fully wearable multimodal interface for real-time VR-interaction, based on concurrent US and inertial (accelerometry) sensing from the forearm and upper arm. The system is built on the WULPUS platform and integrates an end-to-end software framework for real-time acquisition, visualization, and communication with a Unity-based VR environment. A multimodal learning pipeline is introduced for concurrent hand pose and forearm position estimation in 2D space. The interface is evaluated through offline and online experiments with five subjects, during the execution of three functional tasks: cylinder grasping (gross motor) and relocation, marble pinching (fine motor) and relocation, and liquid pouring. For offline experiments, we collect 5 acquisition sessions across multiple days, achieving an average inter-session accuracy across subjects of 80$\pm$6\% for hand pose estimation and 77$\pm$7\% for forearm position estimation. Online validation with minimal fine-tuning (5 min) demonstrates success rates of 92.0$\pm$16.0\%, 88.0$\pm$9.8\%, and 96.0$\pm$8.0\% for the three tasks, respectively. With a power consumption of only 19.9~mW, our system enables more than 2.5 days of continuous use on a small 350 mAh LiPo battery without the need for recharge, enabling truly wearable, multimodal, and functionally meaningful VR interaction.

Problem

• Virtual Reality interaction
• wearable sensing
• A-mode ultrasound
• multimodal system
• real-time interaction

Innovation

• multimodal sensing
• wearable ultrasound
• real-time VR interaction
• hand pose estimation
• inertial sensing