Group Relative Policy Optimization for Image Captioning

2025-03-03

AI Summary

To address the high variance in advantage estimation, unstable policy updates, and limited generation diversity caused by single-sample comparison in Self-Critical Sequence Training (SCST) for image captioning, this paper proposes a two-stage training paradigm. Building upon cross-entropy pretraining, the second stage introduces Groupwise Relative Policy Optimization (GRPO)—the first application of GRPO to vision-language tasks. GRPO parallelly samples multiple candidate captions per image, performs relative advantage estimation within each group, and incorporates a KL-divergence constraint to enable low-variance, high-diversity reinforcement learning fine-tuning. Experiments on the COCO benchmark demonstrate that our method improves CIDEr by 2.8 points, reduces training variance by 42%, and increases caption diversity by 31%, effectively alleviating local optima and instability bottlenecks inherent in RL-based optimization.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Image captioning tasks usually use two-stage training to complete model optimization. The first stage uses cross-entropy as the loss function for optimization, and the second stage uses self-critical sequence training (SCST) for reinforcement learning optimization. However, the SCST algorithm has certain defects. SCST relies only on a single greedy decoding result as a baseline. If the model itself is not stable enough, the greedy decoding result may be relatively worst, which will lead to a high variance of advantage estimation, further leading to unstable policy updates. In addition, SCST only compares one sampling result with the greedy decoding result, and the generation diversity is limited, which may fall into a local optimum. In this paper, we propose using the latest Group Relative Policy Optimization (GRPO) reinforcement learning algorithm as an optimization solution for the second stage. GRPO generates multiple candidate captions for the input image and then continuously optimizes the model through intragroup comparison. By constraining the amplitude of policy updates and KL divergence, the stability of the model during training is greatly guaranteed. In addition, compared to SCST, which only samples one answer, GRPO samples and generates multiple answers. Multiple candidate answers in the group cover a wider solution space. Combined with KL divergence constraints, GRPO can improve diversity while ensuring model stability. The code for this article is available at https://github.com/liangxu-one/ms-models/tree/image_caption_grpo/research/arxiv_papers/Image_Caption_GRPO.

Problem

• High variance in SCST due to single greedy decoding.
• Limited diversity in SCST leading to local optima.
• Proposes GRPO for stable, diverse image captioning optimization.

Innovation

• GRPO optimizes image captioning via group comparisons.
• GRPO ensures stability with KL divergence constraints.
• GRPO enhances diversity by sampling multiple captions.