🤖 AI Summary
This study addresses the challenge of reliably detecting and reconstructing touch inputs on mobile devices from over-the-shoulder videos under conditions where direct screen recording is unavailable. To this end, the authors propose a multimodal touch reconstruction framework that integrates hand anatomical keypoints (via MediaPipe), HSV-based skin-color segmentation, inter-frame difference for motion detection, and Canny edge analysis. The work presents the first systematic evaluation of such an approach in over-the-shoulder surveillance scenarios. Experimental results demonstrate that, in controlled settings, the method achieves an F1 score of 16.7% and a sequence similarity of 3.0%. However, in real-world uncontrolled videos, it suffers from severe false detections—averaging 57 spurious touch points per frame—rendering accurate keystroke sequence recovery impractical and highlighting a critical generalization bottleneck of current methods in real-world applications.
📝 Abstract
Video Intelligence Surveillance (VIDINT) on over-the-shoulder footage is a proposed vector for monitoring human-computer interaction patterns without direct screen recording access. In this paper, we evaluate a Behavioral Intelligence (BEHINT) touch-detection framework designed to reconstruct keystroke events on mobile keypad interfaces from physical finger interactions. Our system integrates four parallel detection modalities: (1) anatomical hand landmarks via MediaPipe, (2) HSV skin color filtering, (3) temporal frame differencing for motion detection, and (4) shape-guided Canny edge analysis. We map relative touch coordinates to a reference screen layout to reconstruct typing sequences. Evaluation on a 120-frame first-person staged video of passcode entry reveals that while MediaPipe and Skin Detection fail to run autonomously due to partial hand occlusion and ambient noise, Motion-Only and Edge-Only configurations achieve F1-scores of 18.5% and 18.2%, respectively. The combined multi-modal configuration achieves an F1-score of 16.7% and a sequence similarity of 3.0% when mapped to the iOS passcode layout. We conduct ablation, resolution decay, noise sensitivity, and proximity threshold tuning to characterize the system's operational envelope. We then audit generalization on 5 real, publicly licensed third-person phone videos and find that the detector emits a median of 57 touch points per frame (peaking at 205), one to three orders of magnitude more than the rate of real taps, because the skin filter responds to the whole hand rather than to fingertip contact. The staged keystroke result does not survive contact with uncontrolled footage; the system does not achieve reliable keystroke reconstruction outside the calibrated staged setting.