BiFingerPose: Bimodal Finger Pose Estimation for Touch Devices

2025-11-21

AI Summary

Existing capacitive imaging methods robustly estimate finger pitch and yaw angles but fail to accurately recover roll angle—particularly beyond 45°—severely limiting interaction precision and robustness. This paper introduces the first dual-modal fusion approach integrating capacitive imaging with under-display fingerprint tile data, coupled with a lightweight deep regression model for accurate, joint estimation of all three degrees of freedom (pitch, yaw, roll). Our key innovation lies in leveraging local deformation patterns and contact orientation cues from fingerprint tiles to resolve geometric ambiguities inherent in single-modal capacitive sensing at large roll angles. User studies demonstrate a 21.3% improvement in pose estimation accuracy, a 2.5× increase in task completion efficiency, and a 23.0% gain in operational accuracy. The method’s practicality and generalizability are further validated in security authentication and natural interaction scenarios.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Finger pose offers promising opportunities to expand human computer interaction capability of touchscreen devices. Existing finger pose estimation algorithms that can be implemented in portable devices predominantly rely on capacitive images, which are currently limited to estimating pitch and yaw angles and exhibit reduced accuracy when processing large-angle inputs (especially when it is greater than 45 degrees). In this paper, we propose BiFingerPose, a novel bimodal based finger pose estimation algorithm capable of simultaneously and accurately predicting comprehensive finger pose information. A bimodal input is explored, including a capacitive image and a fingerprint patch obtained from the touchscreen with an under-screen fingerprint sensor. Our approach leads to reliable estimation of roll angle, which is not achievable using only a single modality. In addition, the prediction performance of other pose parameters has also been greatly improved. The evaluation of a 12-person user study on continuous and discrete interaction tasks further validated the advantages of our approach. Specifically, BiFingerPose outperforms previous SOTA methods with over 21% improvement in prediction performance, 2.5 times higher task completion efficiency, and 23% better user operation accuracy, demonstrating its practical superiority. Finally, we delineate the application space of finger pose with respect to enhancing authentication security and improving interactive experiences, and develop corresponding prototypes to showcase the interaction potential. Our code will be available at https://github.com/XiongjunGuan/DualFingerPose.

Problem

• Estimating comprehensive finger pose on touch devices using bimodal input
• Overcoming capacitive image limitations for large-angle finger pose estimation
• Enabling reliable roll angle prediction not possible with single modality

Innovation

• Bimodal input combines capacitive images and fingerprint patches
• Simultaneously predicts comprehensive finger pose parameters accurately
• Reliably estimates roll angle not achievable with single modality