Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning

2024-11-21

arXiv.org

AI Summary

This study addresses the automation challenge of irrigation-aspiration fluid manipulation in minimally invasive surgery by proposing the first end-to-end vision-driven reinforcement learning (RL) framework. Methodologically, we develop a high-fidelity surgical fluid simulation environment and employ domain randomization to enable efficient sim-to-real transfer onto the da Vinci surgical platform; proximal policy optimization (PPO) is adopted with a custom sparse reward function to achieve coordinated autonomous control of irrigation and aspiration subtasks. Key contributions include: (1) the first end-to-end RL framework for full-cycle irrigation-aspiration control, and (2) a novel paradigm for surgical fluid simulation modeling and transfer learning. Experimental results demonstrate that the system reduces residual contaminants to 2.21 g post-irrigation (vs. 1.90 g manually), leaves 2.24–2.64 g residual fluid post-aspiration, and achieves an overall autonomous reduction of initial 5 g contaminants to 2.42 g.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The irrigation-suction process is a common procedure to rinse and clean up the surgical field in minimally invasive surgery (MIS). In this process, surgeons first irrigate liquid, typically saline, into the surgical scene for rinsing and diluting the contaminant, and then suction the liquid out of the surgical field. While recent advances have shown promising results in the application of reinforcement learning (RL) for automating surgical subtasks, fewer studies have explored the automation of fluid-related tasks. In this work, we explore the automation of both steps in the irrigation-suction procedure and train two vision-based RL agents to complete irrigation and suction autonomously. To achieve this, a platform is developed for creating simulated surgical robot learning environments and for training agents, and two simulated learning environments are built for irrigation and suction with visually plausible fluid rendering capabilities. With techniques such as domain randomization (DR) and carefully designed reward functions, two agents are trained in the simulator and transferred to the real world. Individual evaluations of both agents show satisfactory real-world results. With an initial amount of around 5 grams of contaminants, the irrigation agent ultimately achieved an average of 2.21 grams remaining after a manual suction. As a comparison, fully manual operation by a human results in 1.90 grams remaining. The suction agent achieved 2.64 and 2.24 grams of liquid remaining across two trial groups with more than 20 and 30 grams of initial liquid in the container. Fully autonomous irrigation-suction trials reduce the contaminant in the container from around 5 grams to an average of 2.42 grams, although yielding a higher total weight remaining (4.40) due to residual liquid not suctioned. Further information about the project is available at https://tbs-ualberta.github.io/CRESSim/.

Problem

• Automating surgical irrigation and suction using reinforcement learning
• Developing simulated environments for training vision-based RL agents
• Transferring trained agents from simulation to real-world surgical tasks

Innovation

• Reinforcement learning for surgical irrigation-suction automation
• Simulated surgical environments with fluid rendering
• Domain randomization for real-world transfer