LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation

2025-08-05

AI Summary

Existing text-to-image (T2I) models exhibit significant limitations in fine-grained control—such as text rendering, human pose alignment, and complex compositional reasoning—as well as deep semantic consistency. To address these challenges, we propose LumiGen, an iterative T2I generation framework grounded in vision-language models (VLMs). Its core innovation is a closed-loop feedback mechanism endowed with “visual critique” capability: leveraging VLM-driven intelligent prompt parsing and enhancement, coupled with multi-round visual feedback refinement, to enable precise, stepwise control over the generation process. LumiGen end-to-end integrates diffusion models, VLMs, and iterative optimization modules. Evaluated on the LongBench-T2I benchmark, it achieves a mean score of 3.08—substantially outperforming prior methods—particularly excelling in textual accuracy and pose fidelity, two critical dimensions of fine-grained semantic alignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Text-to-Image (T2I) generation has made significant advancements with diffusion models, yet challenges persist in handling complex instructions, ensuring fine-grained content control, and maintaining deep semantic consistency. Existing T2I models often struggle with tasks like accurate text rendering, precise pose generation, or intricate compositional coherence. Concurrently, Vision-Language Models (LVLMs) have demonstrated powerful capabilities in cross-modal understanding and instruction following. We propose LumiGen, a novel LVLM-enhanced iterative framework designed to elevate T2I model performance, particularly in areas requiring fine-grained control, through a closed-loop, LVLM-driven feedback mechanism. LumiGen comprises an Intelligent Prompt Parsing & Augmentation (IPPA) module for proactive prompt enhancement and an Iterative Visual Feedback & Refinement (IVFR) module, which acts as a "visual critic" to iteratively correct and optimize generated images. Evaluated on the challenging LongBench-T2I Benchmark, LumiGen achieves a superior average score of 3.08, outperforming state-of-the-art baselines. Notably, our framework demonstrates significant improvements in critical dimensions such as text rendering and pose expression, validating the effectiveness of LVLM integration for more controllable and higher-quality image generation.

Problem

• Enhancing fine-grained control in text-to-image generation
• Improving text rendering and pose generation accuracy
• Ensuring deep semantic consistency in generated images

Innovation

• LVLM-enhanced iterative framework for T2I
• Closed-loop LVLM-driven feedback mechanism
• Intelligent Prompt Parsing & Augmentation module