Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting

2026-04-22

AI Summary

This work addresses the challenge of inefficient training in vision-language models for medical report generation, which stems from the scarcity of high-quality annotated data. To mitigate this limitation, the authors propose a token-level reweighted loss function grounded in clinical importance, replacing the conventional cross-entropy loss to dynamically emphasize semantically critical tokens during training. This approach requires no architectural modifications and significantly enhances model performance and data efficiency under low-data regimes. Experimental results on ophthalmic report generation demonstrate that the proposed method achieves comparable report quality to baseline models using only 10% of the training data, consistently exhibiting superior sample efficiency across varying dataset sizes.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Training vision-language models (VLMs) for medical report generation is often hindered by the scarcity of high-quality annotated data. This work evaluates the use of a weighted loss function to improve data efficiency. Compared to standard cross-entropy loss, which treats all token prediction errors equally, the reweighted loss shifts the focus to semantically salient tokens with outsized clinical importance. In experiments on ophthalmological report generation, we show that this simple method improves efficiency across multiple data scales, achieving similar report quality with up to ten times less training data.

Problem

• medical report generation
• sample efficiency
• vision-language models
• data scarcity
• token prediction

Innovation

• token reweighting
• medical report generation
• sample efficiency
• vision-language models
• weighted loss