Autonomous Skeletal Landmark Localization towards Agentic C-Arm Control

2026-04-20

AI Summary

This study addresses the frequent need for manual intervention in C-arm repositioning during emergency procedures, which often delays treatment. The authors propose a novel approach that leverages a fine-tuned multimodal large language model (MLLM) to autonomously localize skeletal landmarks by integrating synthetic and real X-ray data during training. Incorporating clinical feedback enables dynamic C-arm navigation. Evaluated on two datasets, the method achieves landmark localization accuracy comparable to state-of-the-art deep learning models while demonstrating superior reasoning and spatial awareness capabilities. These attributes facilitate error correction and sequential adjustments, substantially enhancing procedural automation in fluoroscopic guidance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Purpose: Automated C-arm positioning ensures timely treatment in patients requiring emergent interventions. When a conventional Deep Learning (DL) approach for C-arm control fails, clinicians must revert to manual operation, resulting in additional delays. Consequently, an agentic C-arm control framework based on multimodal large language models (MLLMs) is highly desirable, as it can incorporate clinician feedback and use reasoning to make adjustments toward more accurate positioning. Skeletal landmark localization is essential for C-arm control, and we investigate adapting MLLMs for autonomous landmark localization. Methods: We used an annotated synthetic X-ray dataset and a real X-ray dataset. Each X-ray in both datasets is paired with several skeletal landmarks. We fine-tuned two MLLMs and tasked them with retrieving the closest landmarks from each X-ray. Quantitative evaluations of landmark localization were performed and compared against a leading DL approach. We further conducted qualitative experiments demonstrating: (1) how an MLLM can correct an initially incorrect prediction through reasoning, and (2) how the MLLM can sequentially navigate the C-arm toward a target location. Results: On both datasets, fine-tuned MLLMs demonstrate competitive performance across all localization tasks when compared with the DL approach. In the qualitative experiments, the MLLMs provide evidence of reasoning and spatial awareness. Conclusion: This study shows that fine-tuned MLLMs achieve accurate skeletal landmark localization and hold promise for agentic autonomous C-arm control. Our code is available athttps://github.com/marszzibros/C-arm-localization-LLMs.git

Problem

• skeletal landmark localization
• C-arm control
• autonomous positioning
• multimodal large language models
• emergency intervention

Innovation

• multimodal large language models
• skeletal landmark localization
• agentic control
• C-arm navigation
• autonomous medical imaging