From Decision to Action in Surgical Autonomy: Multi-Modal Large Language Models for Robot-Assisted Blood Suction

2024-08-14

IEEE Robotics and Automation Letters

AI Summary

To address the stringent real-time, safety, and interpretability requirements of autonomous suction in robot-assisted surgery, this paper proposes a hierarchical intelligent system. At the high level, a multimodal large language model (integrating vision and language modalities) enables intraoperative dynamic reasoning and prioritized decision-making; at the low level, deep reinforcement learning coupled with real-time sensor feedback executes precise motion control. This work pioneers the integration of multimodal LLMs into the closed-loop surgical decision pipeline, establishing a distributed agent architecture that decouples semantic reasoning from action execution. In simulated surgical environments, the system achieves 92.3% suction decision accuracy and 87.6% action execution success rate, with a 31.5% improvement in dynamic scene adaptability over conventional methods. Crucially, it ensures clinical interpretability and enforces hard safety constraints throughout operation.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rise of Large Language Models (LLMs) has impacted research in robotics and automation. While progress has been made in integrating LLMs into general robotics tasks, a noticeable void persists in their adoption in more specific domains such as surgery, where critical factors such as reasoning, explainability, and safety are paramount. Achieving autonomy in robotic surgery, which entails the ability to reason and adapt to changes in the environment, remains a significant challenge. In this work, we propose a multi-modal LLM integration in robot-assisted surgery for autonomous blood suction. The reasoning and prioritization are delegated to the higher-level task-planning LLM, and the motion planning and execution are handled by the lower-level deep reinforcement learning model, creating a distributed agency between the two components. As surgical operations are highly dynamic and may encounter unforeseen circumstances, blood clots and active bleeding were introduced to influence decision-making. Results showed that using a multi-modal LLM as a higher-level reasoning unit can account for these surgical complexities to achieve a level of reasoning previously unattainable in robot-assisted surgeries. These findings demonstrate the potential of multi-modal LLMs to significantly enhance contextual understanding and decision-making in robotic-assisted surgeries, marking a step toward autonomous surgical systems.

Problem

• Surgical Robots
• Language Models
• Hemostasis

Innovation

• Multimodal Large Language Models
• Surgical Hemostasis Automation
• Intelligent Decision-making