AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward

2026-05-12

AI Summary

This work addresses the limitations of unified multimodal models (UMMs) in multimodal generation under cold-start-free settings and their difficulty in implicit intent reasoning. To overcome these challenges, the authors propose a Grouped Relative Policy Optimization (GRPO) method augmented with a self-reflection mechanism. Central to this approach is the Decomposed Verifiable Reward (DVReward), which leverages large language models to decompose complex instructions into atomic semantic and quality sub-problems, enabling multimodal large models to provide stable and interpretable feedback signals for autonomous alignment and refinement of generated content. The proposed method achieves significant performance gains across multiple benchmarks—including GenEval, TIIF-Bench, DPG-Bench, and WISE—and demonstrates strong generalization in editing tasks on GEdit despite no explicit training on such tasks.

Abstract

In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without an additional cold-start stage. Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation. Project page: https://huangrh99.github.io/AlphaGRPO/

Problem

• Self-Reflective Multimodal Generation
• Unified Multimodal Models
• Implicit Intent Inference
• Output Misalignment Correction
• Stable Supervision

Innovation

• Decompositional Verifiable Reward
• Self-Reflective Generation
• Group Relative Policy Optimization
• Unified Multimodal Models
• Multimodal Reinforcement Learning