Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

2026-03-10

AI Summary

Existing unified vision-language models struggle to generate interleaved multimodal content, limiting their applicability in tasks such as visual storytelling and step-by-step reasoning. This work proposes a reinforcement learning–based post-training approach that endows models with high-quality multimodal interleaved generation capabilities without requiring large-scale interleaved data. The key innovation lies in the first extension of Group Relative Policy Optimization (GRPO) to multimodal settings, complemented by a hybrid reward mechanism that integrates text relevance, image-text alignment, and structural fidelity, along with process-level rewards to jointly optimize text and image generation. Evaluated on the MMIE and InterleavedBench benchmarks, the method significantly improves generation quality, coherence, and structural faithfulness, demonstrating its effectiveness and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

Problem

• multimodal interleaved generation
• vision-language models
• visual storytelling
• step-by-step visual reasoning
• unified multimodal generation

Innovation

• multimodal interleaved generation
• Group Relative Policy Optimization
• reinforcement learning
• unified vision-language models
• hybrid reward