Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training

2026-06-19

AI Summary

This work addresses the trade-off between performance and diversity in autoregressive text-to-image (T2I) generation, where performance gains often come at the cost of reduced output variety. Existing GRPO methods employ a fixed treatment of divergence from the reference policy, limiting alignment efficacy. Within a unified f-divergence framework, this study systematically analyzes the effects of forward KL, reverse KL, and Jensen–Shannon (JS) divergences on GRPO policy optimization. It introduces JS divergence into autoregressive T2I alignment for the first time, theoretically demonstrating that JS divergence mitigates the uniformity bias of the reference policy and curbs excessive deviation through token-level update shaping. Experiments on LlamaGen and Janus-7B show that JS divergence achieves state-of-the-art or highly competitive results across most metrics while effectively balancing generation quality and diversity.

Abstract

Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.

Problem

• text-to-image generation
• reinforcement learning
• policy divergence
• generation diversity
• GRPO

Innovation

• GRPO
• f-divergence
• JS divergence
• autoregressive text-to-image
• policy optimization