Magical Touch: Transforming Raw Capacitive Streams into Expressive Hand-Touchscreen Interaction

2026-05-12

AI Summary

This work addresses the limitation of conventional touchscreen interaction, which predominantly relies on fingertip input and overlooks rich information from larger-area contacts such as the palm, thereby constraining the expressive potential of hand-based interaction. Breaking from this paradigm, the study introduces a novel approach that leverages raw capacitive sensing data to reconstruct hand geometry and contact intensity in real time, directly integrating these cues into the interaction pipeline. By combining physics engine–driven modeling with gesture recognition techniques, the system enables natural and continuous hand gestures on standard touchscreen devices. Evaluations through game prototypes—spanning single-user, multi-user collaborative, and pressure-sensitive modes—demonstrate that the method significantly enhances immediate feedback responsiveness to hand posture and applied force, thereby expanding the design space for touch interaction and improving both immersion and expressiveness.

Abstract

Modern touchscreens utilize capacitive sensing technology to enable precise and robust multi-touch interaction. However, the broader expressive potential of the human hand remains underutilized, since most existing methods directly filter out larger-area hand-screen contact. This paper introduces Magical Touch, an interaction method based on raw capacitive sensing data. By directly integrating raw touchscreen sensor data into the interaction loop, our method allows users to interact with the screen naturally and efficiently using arbitrary hand gestures on existing touchscreen devices. To demonstrate the feasibility and expressive capacity of this approach, we implement a physics-based interactive game featuring single-player, multiplayer collaborative, and pressure-sensitive modes. These scenarios showcase how digital objects can respond in real-time to both the geometry and contact intensity of the user's hand. Our results indicate that leveraging raw capacitive data can expand the design space of touchscreen interaction, offering an embodied and continuous interaction paradigm beyond existing fingertip-based approaches.

Problem

• capacitive sensing
• touchscreen interaction
• hand gesture
• expressive interaction
• raw sensor data

Innovation

• raw capacitive sensing
• expressive touch interaction
• hand geometry
• pressure-sensitive input
• embodied interaction