LLMs Enable Context-Aware Augmented Reality in Surgical Navigation

2024-12-21

AI Summary

Current pancreatic surgical navigation systems suffer from limited intelligence and usability, rigid semantic constraints in conventional voice interfaces, and high cognitive load. Method: We propose the first large language model (LLM)-augmented augmented reality (AR) voice navigation system for pancreatic surgery, deployed on wearable AR hardware to enable real-time intraoperative visualization. The system leverages LLM-driven, context-aware natural language understanding and reasoning—moving beyond keyword-matching voice control. Evaluation employs a mixed-method approach (simulation tasks + live surgeries) grounded in human factors engineering. Results: The system significantly reduces task completion time (p < 0.01) and lowers NASA-TLX cognitive workload scores (mean reduction: 32%). Surgeon qualitative feedback indicates marked improvements in intuitiveness, intraoperative decision support, and user preference over baseline systems. This work pioneers deep LLM integration into a closed-loop AR voice interaction framework for surgery, establishing a novel paradigm for intelligent navigation in high-complexity minimally invasive procedures.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Wearable Augmented Reality (AR) technologies are gaining recognition for their potential to transform surgical navigation systems. As these technologies evolve, selecting the right interaction method to control the system becomes crucial. Our work introduces a voice-controlled user interface (VCUI) for surgical AR assistance systems (ARAS), designed for pancreatic surgery, that integrates Large Language Models (LLMs). Employing a mixed-method research approach, we assessed the usability of our LLM-based design in both simulated surgical tasks and during pancreatic surgeries, comparing its performance against conventional VCUI for surgical ARAS using speech commands. Our findings demonstrated the usability of our proposed LLM-based VCUI, yielding a significantly lower task completion time and cognitive workload compared to speech commands. Additionally, qualitative insights from interviews with surgeons aligned with the quantitative data, revealing a strong preference for the LLM-based VCUI. Surgeons emphasized its intuitiveness and highlighted the potential of LLM-based VCUI in expediting decision-making in surgical environments.

Problem

• Surgical Navigation Systems
• Intelligence Enhancement
• Usability Improvement

Innovation

• Large Language Models
• Voice-Controlled User Interface
• Pancreatic Surgery Navigation