🤖 AI Summary
This work addresses the challenge of hallucinations in large vision-language models that compromise clinical reliability in medical report generation. To mitigate this issue, the authors propose the Knowledge-Enhanced Report Generation with Retrieval and Mitigation (KERM) framework, which first retrieves relevant lesion-related knowledge using MedCLIP, then employs a context-aware filtering module to select knowledge consistent with the patient’s imaging findings and medical history, and finally integrates a fine-grained reinforcement learning reward mechanism to guide the model toward generating accurate, evidence-based medical descriptions. Experimental results on the IU-Xray and MIMIC-CXR datasets demonstrate that KERM significantly reduces hallucination rates while improving both clinical accuracy and overall report quality.
📝 Abstract
In the realm of medical report generation (MRG), the integration of natural language processing has emerged as a vital tool to alleviate the workload of radiologists. Despite the impressive capabilities demonstrated by large vision language models (LVLMs) in understanding natural language, their susceptibility to generating plausible yet inaccurate claims, known as ``hallucinations'', raises concerns-especially in the nuanced and critical field of medical. In this work, we introduce a framework, \textbf{K}nowledge-\textbf{E}nhanced with Fine-Grained \textbf{R}einforced Rewards \textbf{M}edical Report Generation (KERM), to tackle the issue. Our approach refines the input to the LVLM by first utilizing MedCLIP for knowledge retrieval, incorporating relevant lesion fact sentences from a curated knowledge corpus. We then introduce a novel purification module to ensure the retrieved knowledge is contextually relevant to the patient's clinical context. Subsequently, we employ fine-grained rewards to guide these models in generating highly supportive and clinically relevant descriptions, ensuring the alignment of model's outputs with desired behaviors. Experimental results on IU-Xray and MIMIC-CXR datasets validate the effectiveness of our approach in mitigating hallucinations and enhancing report quality.