HaloTouch: Using IR Multi-path Interference to Support Touch Interactions With General Surfaces

2025-03-03

AI Summary

Existing depth-camera-based touch systems suffer from high detection error rates, significant latency, and limited hover distance. To address these limitations, this work proposes a universal touch interaction method that requires no surface modification or user-worn devices. We introduce the first modeling of infrared multipath interference—naturally occurring in commercial time-of-flight (ToF) cameras—as an exploitable tactile signal source. Our approach integrates motion-to-photon delay optimization, a lightweight user-adaptive short-term calibration, and inversion of the pressure-deformation relationship to enable high-accuracy contact detection on arbitrary material surfaces, sub-millimeter hover sensing, and continuous pressure estimation. Experiments demonstrate a contact detection accuracy of 99.2%, an end-to-end latency of only 150 ms, and a text input speed of 26.3 adjusted words per minute (AWPM). This work redefines the conventional application boundary of ToF cameras in interactive perception, establishing a new paradigm for seamless, ubiquitous touch interaction.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Sensing touch on arbitrary surfaces has long been a goal of ubiquitous computing, but often requires instrumenting the surface. Depth camera-based systems have emerged as a promising solution for minimizing instrumentation, but at the cost of high touch-down detection error rates, high touch latency, and high minimum hover distance, limiting them to basic tasks. We developed HaloTouch, a vision-based system which exploits a multipath interference effect from an off-the-shelf time-of-flight depth camera to enable fast, accurate touch interactions on general surfaces. HaloTouch achieves a 99.2% touch-down detection accuracy across various materials, with a motion-to-photon latency of 150 ms. With a brief (20s) user-specific calibration, HaloTouch supports millimeter-accurate hover sensing as well as continuous pressure sensing. We conducted a user study with 12 participants, including a typing task demonstrating text input at 26.3 AWPM. HaloTouch shows promise for more robust, dynamic touch interactions without instrumenting surfaces or adding hardware to users.

Problem

• Enables touch interactions on arbitrary surfaces without instrumentation.
• Reduces touch latency and improves touch-down detection accuracy.
• Supports millimeter-accurate hover and continuous pressure sensing.

Innovation

• Exploits IR multipath interference for touch sensing
• Achieves 99.2% touch-down detection accuracy
• Supports millimeter-accurate hover and pressure sensing