CRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs

2024-12-31

AI Summary

To address low efficiency, poor consistency, and diagnostic instability in manual chest X-ray (CXR) report generation under high clinical workload, this paper proposes a dual-module synergistic framework for intelligent CXR analysis. First, a region-aware GPT-2 model generates structured radiology reports by integrating Faster R-CNN–based lesion localization with binary critical-region filtering. Second, an unsupervised, CLIP-driven multimodal contrastive learning module performs CXR classification using unlabeled image–text pairs. Critically, the framework introduces a novel “generation–classification” joint optimization paradigm to eliminate information fragmentation between modules. Experiments demonstrate that the proposed method outperforms GPT-4o in report generation (superior BLEU-2/3/4 and ROUGE-L scores) and achieves significantly higher AUC and accuracy than current state-of-the-art models in CXR classification. This unified framework delivers an efficient, robust, and clinically deployable solution for radiology workflow automation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The complexity of stacked imaging and the massive number of radiographs make writing radiology reports complex and inefficient. Even highly experienced radiologists struggle to maintain accuracy and consistency in interpreting radiographs under prolonged high-intensity work. To address these issues, this work proposes the CRRG-CLIP Model (Chest Radiology Report Generation and Radiograph Classification Model), an end-to-end model for automated report generation and radiograph classification. The model consists of two modules: the radiology report generation module and the radiograph classification module. The generation module uses Faster R-CNN to identify anatomical regions in radiographs, a binary classifier to select key regions, and GPT-2 to generate semantically coherent reports. The classification module uses the unsupervised Contrastive Language Image Pretraining (CLIP) model, addressing the challenges of high-cost labelled datasets and insufficient features. The results show that the generation module performs comparably to high-performance baseline models on BLEU, METEOR, and ROUGE-L metrics, and outperformed the GPT-4o model on BLEU-2, BLEU-3, BLEU-4, and ROUGE-L metrics. The classification module significantly surpasses the state-of-the-art model in AUC and Accuracy. This demonstrates that the proposed model achieves high accuracy, readability, and fluency in report generation, while multimodal contrastive training with unlabelled radiograph-report pairs enhances classification performance.

Problem

• Radiology Report
• Efficiency
• Accuracy

Innovation

• CRRG-CLIP
• GPT-2
• CLIP