Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation

2026-08-09

AI Summary

Existing medical report generation methods often conflate clinical facts with linguistic style during preference optimization and lack explicit vision–language alignment, leading to factual inaccuracies. To address this, this work proposes the DPO-Clin framework, which constructs linguistically aligned preference pairs through entity-level clinical diagnosis (ECD) and introduces a retrieval-augmented multimodal direct preference optimization mechanism (M²DPO). By incorporating counterfactual preference data, M²DPO mitigates the risk of high-uncertainty predictions. This approach achieves, for the first time, cross-modal consistency optimization guided by clinical findings. Evaluated on MIMIC-CXR, IU X-Ray, and endoscopic datasets, DPO-Clin significantly outperforms baseline models, demonstrating state-of-the-art performance in both clinical accuracy and cross-architecture generalization.

Abstract

Despite significant advances in Medical Report Generation (MRG), the reliability remains constrained by the prevalence of factual errors. While Direct Preference Optimization (DPO) has emerged as a promising post-training paradigm to enhance the performance of Supervised Fine-Tuned (SFT) MRG models, existing DPO-based MRG methods typically adopt a naive preference construction that directly pairs model-generated reports with ground truth reports. This strategy inadvertently entangles critical clinical findings with clinically irrelevant linguistic characteristics, and fundamentally lacks explicit vision-language alignment. To address these challenges, we propose DPO-Clin, a novel post-training framework that focuses preference optimization on clinical findings and cross-modal alignment. First, we introduce the Entity-level Clinical Diagnostic (ECD) module to perform a precise entity-level factual diagnosis. ECD guides the generation of linguistically-aligned report preference pairs, isolating clinical discrepancies from linguistic variations. Second, to achieve fine-grained cross-modal alignment, we develop M$^2$DPO, a retrieval-augmented multi-modal DPO variant that enforces textual preference inversion triggered by visual context switches. Third, we locate correct yet highly uncertain predicted entities and apply counterfactual modifications to construct targeted preference data for latent risk mitigation, thereby further enhancing the model reliability. Extensive experiments on two public chest X-ray datasets (MIMIC-CXR and IU X-Ray) and an in-house endoscopy dataset demonstrate that DPO-Clin significantly improves the SFT baselines on clinical-aware metrics. Furthermore, it achieves superior performance over existing DPO-based MRG methods, exhibiting robust generalizability across distinct baseline architectures and diverse medical imaging modalities.

Problem

• Medical Report Generation
• Direct Preference Optimization
• Clinical Factual Errors
• Vision-Language Alignment
• Preference Construction

Innovation

• Direct Preference Optimization
• Clinical Report Generation
• Vision-Language Alignment
• Entity-level Diagnosis
• Counterfactual Preference