OrthoInsight: Rib Fracture Diagnosis and Report Generation Based on Multi-Modal Large Models

2025-07-18

AI Summary

To address the time-consuming, error-prone manual diagnosis and inefficient report generation for rib fractures in CT imaging, this paper proposes an end-to-end multimodal diagnostic framework. Methodologically, it integrates YOLOv9 for precise rib fracture localization, incorporates a medical knowledge graph to enhance clinical reasoning, and fine-tunes LLaVA to build an image–text joint generation model that automatically produces structured diagnostic reports from CT scans. Its key innovation lies in the first deep coupling of object detection, domain-specific knowledge retrieval, and multimodal large language model generation—significantly improving clinical applicability. Evaluated on 28,675 annotated CT cases, the framework achieves an average score of 4.28/5.0 across four clinical metrics: diagnostic accuracy, report completeness, logical coherence, and clinical guidance value—outperforming general-purpose large models such as GPT-4.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The growing volume of medical imaging data has increased the need for automated diagnostic tools, especially for musculoskeletal injuries like rib fractures, commonly detected via CT scans. Manual interpretation is time-consuming and error-prone. We propose OrthoInsight, a multi-modal deep learning framework for rib fracture diagnosis and report generation. It integrates a YOLOv9 model for fracture detection, a medical knowledge graph for retrieving clinical context, and a fine-tuned LLaVA language model for generating diagnostic reports. OrthoInsight combines visual features from CT images with expert textual data to deliver clinically useful outputs. Evaluated on 28,675 annotated CT images and expert reports, it achieves high performance across Diagnostic Accuracy, Content Completeness, Logical Coherence, and Clinical Guidance Value, with an average score of 4.28, outperforming models like GPT-4 and Claude-3. This study demonstrates the potential of multi-modal learning in transforming medical image analysis and providing effective support for radiologists.

Problem

• Automates rib fracture diagnosis from CT scans to reduce errors
• Generates clinical reports by combining visual and textual data
• Improves diagnostic accuracy and efficiency for musculoskeletal injuries

Innovation

• YOLOv9 model for fracture detection
• Medical knowledge graph for clinical context
• Fine-tuned LLaVA model for report generation