InstanceGen: Image Generation with Instance-level Instructions

2025-05-08

AI Summary

Current text-to-image models struggle to accurately model object cardinality, instance-level attributes, and spatial relationships specified in complex prompts, resulting in low semantic alignment between generated images and textual descriptions. To address this, we propose a structure-aware end-to-end alignment framework that uniquely leverages intrinsic image structures—such as segmentation masks and layout heatmaps—extracted from the generated image itself as fine-grained initialization signals. These structural cues are tightly coupled with instance-level instructions parsed by a large language model (LLM) to guide the diffusion process. Our method integrates vision-language joint reasoning, LLM-driven instruction understanding, and image-structure inversion. Evaluated on multi-instance composition benchmarks, our approach achieves significant gains in prompt fidelity: object counting accuracy improves by 32%, while attribute and spatial layout consistency reach state-of-the-art performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite rapid advancements in the capabilities of generative models, pretrained text-to-image models still struggle in capturing the semantics conveyed by complex prompts that compound multiple objects and instance-level attributes. Consequently, we are witnessing growing interests in integrating additional structural constraints, %leveraging additional structural inputs typically in the form of coarse bounding boxes, to better guide the generation process in such challenging cases. In this work, we take the idea of structural guidance a step further by making the observation that contemporary image generation models can directly provide a plausible emph{fine-grained} structural initialization. We propose a technique that couples this image-based structural guidance with LLM-based instance-level instructions, yielding output images that adhere to all parts of the text prompt, including object counts, instance-level attributes, and spatial relations between instances.

Problem

• Text-to-image models fail with complex multi-object prompts
• Existing methods lack fine-grained structural guidance
• Need better adherence to instance-level attributes and relations

Innovation

• Uses image-based structural guidance
• Integrates LLM-based instance-level instructions
• Enhances adherence to complex text prompts