From Scanning Guidelines to Action: A Robotic Ultrasound Agent with LLM-Based Reasoning

2026-03-15

AI Summary

This work proposes the first large language model (LLM)-based agent framework for robotic ultrasound, addressing the limitations of conventional systems that rely on fixed protocols and task-specific models and thus struggle to meet diverse clinical scanning demands. By interpreting clinical ultrasound guidelines, dynamically invoking appropriate tools, and optimizing decision-making through reinforcement learning, the framework enables interpretable and generalizable autonomous scanning. The approach demonstrates high accuracy in reasoning and tool invocation across ten distinct ultrasound guidelines and exhibits practical cross-task feasibility and generalization capability in robotic scans of the gallbladder, spine, and kidneys.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Robotic ultrasound offers advantages over free-hand scanning, including improved reproducibility and reduced operator dependency. In clinical practice, US acquisition relies heavily on the sonographer's experience and situational judgment. When transferring this process to robotic systems, such expertise is often encoded explicitly through fixed procedures and task-specific models, yielding pipelines that can be difficult to adapt to new scanning tasks. In this work, we propose a unified framework for autonomous robotic US scanning that leverages a LLM-based agent to interpret US scanning guidelines and execute scans by dynamically invoking a set of provided software tools. Instead of encoding fixed scanning procedures, the LLM agent retrieves and reasons over guideline steps from scanning handbooks and adapts its planning decisions based on observations and the current scanning state. This enables the system to handle variable and decision-dependent workflows, such as adjusting scanning strategies, repeating steps, or selecting the appropriate next tool call in response to image quality or anatomical findings. Because the reasoning underlying tool selection is also critical for transparent and trustworthy planning, we further fine tune the LLM agent using a RL based strategy to improve both its reasoning quality and the correctness of tool selection and parameterization, while maintaining robust generalization to unseen guidelines and related tasks. We first validate the approach via verbal execution on 10 US scanning guidelines, assessing reasoning as well as tool selection and parameterization, and showing the benefit of RL fine tuning. We then demonstrate real world feasibility on robotic scanning of the gallbladder, spine, and kidney. Overall, the framework follows diverse guidelines and enables reliable autonomous scanning across multiple anatomical targets within a unified system.

Problem

• robotic ultrasound
• autonomous scanning
• LLM-based reasoning
• scanning guidelines
• adaptive workflow

Innovation

• LLM-based agent
• robotic ultrasound
• autonomous scanning
• reinforcement learning fine-tuning
• dynamic tool invocation