Unlocking Aha Moments via Reinforcement Learning: Advancing Collaborative Visual Comprehension and Generation

2025-06-02

AI Summary

Existing multimodal large language models (MLLMs) suffer from a fundamental decoupling between visual understanding and generation capabilities, hindering their co-evolution and lacking interpretable, self-reflective generation mechanisms. To address this, we propose a novel understanding-generation co-evolution paradigm: modeling image generation as an iterative, chain-of-thought (CoT)-guided self-reflection process, and—crucially—introducing reinforcement learning (RL) into visual generation for the first time to elicit “aha moments.” Our method employs a two-stage training framework: supervised fine-tuning to bootstrap CoT-based reasoning capability, followed by Proximal Policy Optimization (PPO) to balance exploration and exploitation. Furthermore, we integrate MLLMs with a CoT-driven cross-modal alignment mechanism. Evaluated on text-to-image generation and image editing tasks, our approach achieves state-of-the-art performance while substantially improving semantic image assessment and visual understanding—unifying enhanced controllability, interpretability, and comprehension in generative vision systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent endeavors in Multimodal Large Language Models (MLLMs) aim to unify visual comprehension and generation. However, these two capabilities remain largely independent, as if they are two separate functions encapsulated within the same model. Consequently, visual comprehension does not enhance visual generation, and the reasoning mechanisms of LLMs have not been fully integrated to revolutionize image generation. In this paper, we propose to enable the collaborative co-evolution of visual comprehension and generation, advancing image generation into an iterative introspective process. We introduce a two-stage training approach: supervised fine-tuning teaches the MLLM with the foundational ability to generate genuine CoT for visual generation, while reinforcement learning activates its full potential via an exploration-exploitation trade-off. Ultimately, we unlock the Aha moment in visual generation, advancing MLLMs from text-to-image tasks to unified image generation. Extensive experiments demonstrate that our model not only excels in text-to-image generation and image editing, but also functions as a superior image semantic evaluator with enhanced visual comprehension capabilities. Project Page: https://janus-pro-r1.github.io.

Problem

• Unifying visual comprehension and generation in MLLMs
• Integrating LLM reasoning to enhance image generation
• Enabling iterative introspective image generation process

Innovation

• Two-stage training for visual comprehension and generation
• Reinforcement learning enhances exploration-exploitation trade-off
• Unified image generation via iterative introspective process