ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoning

2025-03-25

AI Summary

This paper addresses the limited structured reasoning capability of multimodal large language models (MLLMs) in text-to-image in-context learning (T2I-ICL). To this end, we propose ImageGen-CoT—a novel framework that explicitly integrates chain-of-thought (CoT) reasoning into T2I-ICL, enabling interpretable, intermediate reasoning prior to image generation. Our key contributions are threefold: (1) construction of the first high-quality, automatically curated ImageGen-CoT dataset; (2) design of a T2I-ICL–specific supervised fine-tuning paradigm and an automated CoT data synthesis pipeline; and (3) introduction of a test-time hybrid scaling strategy that jointly leverages multi-chain reasoning and diverse sample generation. Evaluated on the SEED-X benchmark, our method achieves an 80% improvement in T2I-ICL performance, significantly enhancing modeling of complex instructions and context-dependent relationships.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this work, we study the problem of Text-to-Image In-Context Learning (T2I-ICL). While Unified Multimodal LLMs (MLLMs) have advanced rapidly in recent years, they struggle with contextual reasoning in T2I-ICL scenarios. To address this limitation, we propose a novel framework that incorporates a thought process called ImageGen-CoT prior to image generation. To avoid generating unstructured ineffective reasoning steps, we develop an automatic pipeline to curate a high-quality ImageGen-CoT dataset. We then fine-tune MLLMs using this dataset to enhance their contextual reasoning capabilities. To further enhance performance, we explore test-time scale-up strategies and propose a novel hybrid scaling approach. This approach first generates multiple ImageGen-CoT chains and then produces multiple images for each chain via sampling. Extensive experiments demonstrate the effectiveness of our proposed method. Notably, fine-tuning with the ImageGen-CoT dataset leads to a substantial 80% performance gain for SEED-X on T2I-ICL tasks. See our project page at https://ImageGen-CoT.github.io/. Code and model weights will be open-sourced.

Problem

• Enhancing contextual reasoning in text-to-image learning
• Generating structured reasoning steps for image generation
• Improving multimodal LLMs' performance via fine-tuning and scaling

Innovation

• Incorporates Chain-of-Thought reasoning for image generation
• Automatic pipeline curates high-quality reasoning dataset
• Hybrid scaling generates multiple reasoning chains and images