Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light

2025-01-15

AI Summary

Minimally invasive surgical robots face spatial constraints that hinder integration of physical force sensors, impeding real-time, high-precision soft-tissue contact force sensing. Method: This paper proposes a sensorless force estimation approach that fuses single-shot structured-light projection with binocular endoscopic vision to reconstruct high-resolution 3D deformation point clouds of soft tissue, and employs an improved PointNet-based end-to-end neural network to directly map tissue deformation to interaction force. Contribution/Results: The method introduces the novel paradigm of “single-shot structured-light encoding + nonlinear mechanical response modeling,” overcoming accuracy limitations of conventional vision-based force estimation. Experimental validation on multi-stiffness silicone phantoms achieves a mean force estimation error <0.08 N—reaching millinewton-level precision—and demonstrates a threefold improvement over state-of-the-art visual methods, satisfying clinical safety thresholds.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, most existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. Based on this, a modified PointNet-based force estimation method is proposed, which excels in representing the complex mechanical properties of soft tissue. Numerical force interaction experiments are conducted on three silicon materials with different stiffness. The results validate the effectiveness of the proposed scheme.

Problem

• Robot-assisted Minimally Invasive Surgery
• Force Feedback Measurement
• Sensorless Approach

Innovation

• Structural Light
• Deep Learning
• Force Measurement