REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation

2026-07-11

AI Summary

Existing reinforcement learning approaches for chest X-ray report generation suffer from training instability and insufficient exploration due to fixed KL regularization and static reference policies. To address these limitations, this work proposes the REVA-PO framework, which introduces response-weighted regularization to dynamically adjust the strength of KL constraints and incorporates a validation-anchored policy reset mechanism to periodically synchronize with the optimal policy. The framework further employs a three-stage training pipeline—comprising warm-up, classifier-guided fine-tuning, and reinforcement learning—to enhance both stability and generation quality. Evaluated on MIMIC-CXR and IU-Xray benchmarks, REVA-PO achieves new state-of-the-art performance, improving BLEU-4 scores by 5.1% and 3.6%, respectively, and boosting CheXpert F1 and RadGraph F1 by 4.5% and 12.8%.

Abstract

Automated chest X-ray report generation has recently benefited from reinforcement learning (RL) and large language models. However, RL training often suffers from instability or limited exploration due to fixed Kullback-Leibler (KL) regularization and a static reference policy that accumulates KL pressure over time. We propose Response-Weighted and Validation-Anchored Policy Optimization (REVA-PO), a RL framework that stabilizes long-term training via Response-Weighted Regularization (RER) and Validation-Anchored Policy Reset (VAPR). RER dynamically adjusts per-response KL weights based on advantage and reference-policy entropy, relaxing constraints for high-quality responses while tightening them for low-quality ones. Complementarily, VAPR periodically synchronizes the reference and current policies to the best validation checkpoint, resetting accumulated regularization pressure to expand the viable exploration space. To ensure a robust starting point, we employ a three-stage pipeline consisting of warm-up training, classifier-guided supervised fine-tuning, and RL. Extensive evaluations on MIMIC-CXR and IU-Xray demonstrate that REVA-PO sets new state-of-the-art benchmarks in both linguistic quality and clinical accuracy. Notably, BLEU-4 improves by 5.1% on MIMIC-CXR and 3.6% on IU-Xray, while CheXpert F1 and RadGraph F1 scores increase by 4.5% and 12.8%, respectively, over prior leading methods. The code is publicly available at https://github.com/LiGuo12/REVA_PO/.

Problem

• Reinforcement Learning
• Chest X-ray Report Generation
• KL Regularization
• Training Instability
• Policy Optimization

Innovation

• REVA-PO
• Response-Weighted Regularization
• Validation-Anchored Policy Reset
• Reinforcement Learning
• Chest X-ray Report Generation