Wearable Multimodal Human-Machine Interface for Integrated Hand Intentions Decoding in Dynamic Teleoperation

2026-09-07

AI Summary

为解决在光学条件差的远程操作环境中手部意图解码问题，研究开发了一种结合表面肌电图传感器和惯性测量单元的新型可穿戴人机接口，并通过多模态深度学习方法显著提高了手势识别精度、抓握力估计准确度和手姿态解码一致性。

Abstract

Under ubiquitous teleoperation environments with optically challenging conditions, an interface for tele-operated grasping that combines wearability with precise decoding of hand intentions (hand pose, gestures, and grasping force) is essential. Yet, existing interfaces often fall short in meeting these demands, compromising either the diversity of multiple intentions decoding or wearability. To address this, we developed a novel Multiple Intentions Decoding Human-Machine Interface (MI-DHMI) that integrates high-throughput surface electromyography (sEMG) sensors with hand-mounted and forearm-mounted inertial measurement units (IMUs). The developed interface is supported by a unified framework for simultaneous multiple intentions decoding. By employing multimodal deep learning and hardware design with a low noise floor, the decoding framework selectively focuses on the sEMG components that are genuinely associated with finger movements. This effectively reduces decoding errors caused by sEMG variability during unconstrained upper-limb motions, thereby significantly enhancing robustness. Even under unconstrained wrist and forearm motion, the interface achieves a gesture recognition accuracy exceeding 97%, grasping force estimation with $R^2 = 0.95$, and hand pose decoding consistent with the actual hand pose, outperforming baseline devices and algorithms. Ablation studies further validate the effectiveness of the proposed decoding framework. Finally, two online experiments were conducted to validate the device, demonstrating its superior performance in high-stability tasks, including a pouring task and object grasping. The developed interface provides a new solution of a fully wearable, multiple intentions decoding system, offering effective support for ubiquitous teleoperation and contributing to the advancement of human-machine interaction research.

Problem

• Wearable Interface
• Hand Intentions Decoding
• Teleoperation
• Multimodal Human-Machine Interface
• Ubiquitous Teleoperation

Innovation

• surface electromyography (sEMG)
• inertial measurement units (IMUs)
• multimodal deep learning
• multiple intentions decoding
• wearable interface