Surgical Agent Orchestration Platform for Voice-directed Patient Data Interaction

2025-11-10

AI Summary

In da Vinci robotic surgery, surgeons face clinical challenges in hands-free operation and real-time access to multimodal patient data. Method: This paper proposes a hierarchical multi-agent, voice-driven surgical collaboration platform centered on a large language model (LLM) as the orchestrator, integrating automatic speech recognition, semantic understanding, and task orchestration to enable clinical information retrieval, CT image interaction, and 3D anatomical navigation. Contribution/Results: We introduce a novel hierarchical multi-agent architecture and a multi-level evaluation metric (MOEM), significantly enhancing robustness against speech errors and ambiguous commands while supporting autonomous intraoperative task planning and semantic reasoning. Evaluated on 240 real-world surgical voice commands, the system achieves high accuracy and task success rates, demonstrating its feasibility for efficient and reliable human–robot collaboration in minimally invasive robotic surgery.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

In da Vinci robotic surgery, surgeons'hands and eyes are fully engaged in the procedure, making it difficult to access and manipulate multimodal patient data without interruption. We propose a voice-directed Surgical Agent Orchestrator Platform (SAOP) built on a hierarchical multi-agent framework, consisting of an orchestration agent and three task-specific agents driven by Large Language Models (LLMs). These LLM-based agents autonomously plan, refine, validate, and reason to map voice commands into specific tasks such as retrieving clinical information, manipulating CT scans, or navigating 3D anatomical models on the surgical video. We also introduce a Multi-level Orchestration Evaluation Metric (MOEM) to comprehensively assess the performance and robustness from command-level and category-level perspectives. The SAOP achieves high accuracy and success rates across 240 voice commands, while LLM-based agents improve robustness against speech recognition errors and diverse or ambiguous free-form commands, demonstrating strong potential to support minimally invasive da Vinci robotic surgery.

Problem

• Enables voice-controlled data access during hands-free robotic surgery
• Maps voice commands to medical data retrieval and manipulation tasks
• Improves robustness against speech errors and ambiguous commands

Innovation

• Voice-directed multi-agent platform for robotic surgery
• LLM-based agents autonomously map commands to tasks
• Multi-level evaluation metric assesses performance robustness