Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation

2024-07-21

arXiv.org

AI Summary

Radiology report generation suffers from frequent factual inconsistencies, particularly compromising diagnostic reliability in cardiac conditions. To address this, we propose a fact-aware multimodal retrieval-augmented generation (RAG) framework. First, we construct a medical fact graph grounded in RadGraph and design a diagnosis-label-free, fact-driven multimodal contrastive learning paradigm to train a general-purpose retriever. Subsequently, we transfer this fact-aware capability to the generator to enable end-to-end factual consistency optimization. Our method integrates knowledge mining, RAG, and dual evaluation using CheXbert and RadGraph metrics. On two established benchmarks, it achieves +6.5% in F1_CheXbert and +2.0% in F1_RadGraph over state-of-the-art methods, demonstrating significant improvements in both clinical fact completeness and diagnostic accuracy.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal foundation models hold significant potential for automating radiology report generation, thereby assisting clinicians in diagnosing cardiac diseases. However, generated reports often suffer from serious factual inaccuracy. In this paper, we introduce a fact-aware multimodal retrieval-augmented pipeline in generating accurate radiology reports (FactMM-RAG). We first leverage RadGraph to mine factual report pairs, then integrate factual knowledge to train a universal multimodal retriever. Given a radiology image, our retriever can identify high-quality reference reports to augment multimodal foundation models, thus enhancing the factual completeness and correctness of report generation. Experiments on two benchmark datasets show that our multimodal retriever outperforms state-of-the-art retrievers on both language generation and radiology-specific metrics, up to 6.5% and 2% score in F1CheXbert and F1RadGraph. Further analysis indicates that employing our factually-informed training strategy imposes an effective supervision signal, without relying on explicit diagnostic label guidance, and successfully propagates fact-aware capabilities from the multimodal retriever to the multimodal foundation model in radiology report generation.

Problem

• Automate radiology report generation
• Enhance factual accuracy in reports
• Integrate multimodal retrieval for training

Innovation

• Fact-aware multimodal retrieval-augmented pipeline
• Universal multimodal retriever training
• Factually-informed training strategy