Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation

2025-11-20

AI Summary

Existing visual generation methods lack real-time multimodal interaction during generation, hindering dynamic integration of textual reasoning to optimize semantic fidelity and contextual consistency. To address this, we propose Thinking-while-Generating (TwiG), the first framework enabling fine-grained, temporally interleaved coordination between visual generation and textual reasoning—supporting concurrent generation, planning, and reflection. Methodologically, we construct TwiG-50K, a high-quality interleaved reasoning dataset; integrate zero-shot prompting, supervised fine-tuning, and a customized TwiG-GRPO reinforcement learning strategy. Experiments demonstrate substantial improvements in semantic coherence, detail accuracy, and cross-modal alignment, validating the efficacy of co-evolving generation and reasoning. TwiG establishes a novel paradigm for multimodal generative modeling grounded in iterative, bidirectional interaction between vision and language modules.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Recent advances in visual generation have increasingly explored the integration of reasoning capabilities. They incorporate textual reasoning, i.e., think, either before (as pre-planning) or after (as post-refinement) the generation process, yet they lack on-the-fly multimodal interaction during the generation itself. In this preliminary study, we introduce Thinking-while-Generating (TwiG), the first interleaved framework that enables co-evolving textual reasoning throughout the visual generation process. As visual content is progressively generating, textual reasoning is interleaved to both guide upcoming local regions and reflect on previously synthesized ones. This dynamic interplay produces more context-aware and semantically rich visual outputs. To unveil the potential of this framework, we investigate three candidate strategies, zero-shot prompting, supervised fine-tuning (SFT) on our curated TwiG-50K dataset, and reinforcement learning (RL) via a customized TwiG-GRPO strategy, each offering unique insights into the dynamics of interleaved reasoning. We hope this work inspires further research into interleaving textual reasoning for enhanced visual generation. Code will be released at: https://github.com/ZiyuGuo99/Thinking-while-Generating.

Problem

• Integrating textual reasoning during visual generation process
• Enabling co-evolution of reasoning and visual content creation
• Developing interleaved framework for context-aware visual outputs

Innovation

• Interleaving textual reasoning throughout visual generation process
• Using zero-shot prompting, supervised fine-tuning, and reinforcement learning
• Enabling co-evolving reasoning to guide and reflect on generation