TouchFusion: Multimodal Wristband Sensing for Ubiquitous Touch Interactions

2026-02-16

AI Summary

This work proposes a universal surface-based touch interaction method that requires no additional hardware or computer vision, enabling touch recognition on both environmental and body surfaces. By integrating surface electromyography (sEMG), bioimpedance, inertial (IMU), and optical signals from a wrist-worn device—combined with early- and late-stage multimodal fusion strategies—the system achieves state-aware touch detection, basic gesture recognition, and context-adaptive interaction. The approach is validated on a large-scale dataset comprising 100 participants, demonstrating robust generalization across diverse anatomical structures and behavioral patterns. It supports plug-and-play touch interaction on arbitrary surfaces as well as palm-based input, and can be seamlessly integrated with smart glasses to form a pervasive touch interface.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

TouchFusion is a wristband that enables touch interactions on nearby surfaces without any additional instrumentation or computer vision. TouchFusion combines surface electromyography (sEMG), bioimpedance, inertial, and optical sensing to capture multiple facets of hand activity during touch interactions. Through a combination of early and late fusion, TouchFusion enables stateful touch detection on both environmental and body surfaces, simple surface gestures, and tracking functionality for contextually adaptive interfaces as well as basic trackpad-like interactions. We validate our approach on a dataset of 100 participants, significantly exceeding the population size of typical wearable sensing studies to capture a wider variance of wrist anatomies, skin conductivities, and behavioral patterns. We show that TouchFusion can enable several common touch interaction tasks. Using TouchFusion, a wearer can summon a trackpad on any surface, control contextually adaptive interfaces based on where they tap, or use their palm as an always-available touch surface. When paired with smart glasses or augmented reality devices, TouchFusion enables a ubiquitous, contextually adaptive interaction model.

Problem

• touch interaction
• wrist-worn sensing
• ubiquitous computing
• surface electromyography
• contextual interface

Innovation

• multimodal sensing
• touch interaction
• wrist-worn interface
• sensor fusion
• ubiquitous computing