BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

2026-05-08

AI Summary

Existing reinforcement learning–based image captioning methods often optimize a single quality dimension in isolation, leading to imbalanced trade-offs among utility, coverage, and linguistic fluency. This work proposes a balanced multi-objective reinforcement learning framework that, for the first time, integrates continuous reward decoupling with normalization and a length-conditioned reward masking mechanism into multimodal large language models (MLLMs) for image captioning. The approach jointly optimizes utility-aware factual correctness, reference coverage, and language quality. Evaluated on models such as LLaVA and Qwen2.5-VL using the GRPO algorithm, the proposed method achieves substantial improvements in overall captioning performance, with gains of up to +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena.

Abstract

Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that improve downstream question answering while harming fluency, whereas arena-style objectives can favor fluent but generic descriptions with limited usefulness. To address this, we propose a more balanced RL framework that jointly optimizes utility-aware correctness, reference coverage, and linguistic quality. In order to effectively optimize the resulting continuous multi-objective reward formulation, we apply GDPO-style reward-decoupled normalization to continuous-valued captioning rewards and show that it improves performance over vanilla GRPO. Additionally, we introduce length-conditional reward masking, yielding a more suitable length penalty for captioning. Across LLaVA-1.5-7B and Qwen2.5-VL 3B and 7B base models, our method consistently improves caption quality, with peak gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across different models.

Problem

• image captioning
• reinforcement learning
• multimodal large language models
• caption quality
• multi-objective optimization

Innovation

• Balanced Reinforcement Learning
• Multimodal Large Language Models
• Multi-objective Reward Optimization
• Reward Decoupled Normalization
• Length-conditional Reward Masking