Touchless Intraoperative Image Access System Based on Vision-Based Hand Tracking

2026-04-27

AI Summary

This study addresses the need for touchless interaction with medical images during surgery to maintain sterility and procedural continuity. The authors propose an intraoperative hand-gesture interface using a single RGB camera, leveraging MediaPipe Hands for real-time 2.5D hand keypoint estimation. Intuitive gestures are mapped to translation, rotation, and scaling commands that drive a PyVista-based image viewer, enabling natural and seamless interaction. The system requires no additional hardware, user calibration, or specialized training, and its architecture is decoupled from the underlying visualization software, offering low cost and easy deployment. Experimental results demonstrate that the approach achieves real-time performance, low latency, and high stability, fulfilling the demands of smooth intraoperative interaction and confirming its clinical feasibility.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Manipulation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Security and Privacy: Data transparency and provenance
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Touchless interaction with medical images is becoming increasingly important in the surgical field, where sterility and continuity of the operational workflow are essential requirements. This work presents a vision-based system for intraoperative navigation of medical images through hand gestures acquired using a single RGB camera. Unlike many existing solutions, the system does not require additional hardware or user-specific training. Hand tracking is performed in real time using MediaPipe Hands, which provides a 2.5D estimation of hand landmarks. Simple and intuitive gestures are then mapped into translation, rotation, and zoom commands, enabling continuous and natural interaction with the image viewer. The system architecture is independent from the visualization software and, for implementation simplicity, in this study it was integrated with PyVista. Performance was evaluated through frame-level logging and quantitative analysis of latency, stability, and interaction robustness metrics. Experimental results highlight real-time behavior, with reduced latencies and stable control, in line with the requirements of fluid interaction. The system demonstrates the feasibility of a low-cost touchless solution for intraoperative access to medical images, laying the groundwork for future clinical evaluations.

Problem

• touchless interaction
• intraoperative image access
• hand gesture
• medical imaging
• sterility

Innovation

• touchless interaction
• vision-based hand tracking
• intraoperative image navigation
• MediaPipe Hands
• gesture-based control