Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation

2025-07-05

AI Summary

Traditional text-to-image generation relies on large-scale manually annotated image-text pairs and struggles to precisely model fine-grained attributes and spatial relationships in complex prompts. To address this, we propose Hi-SSLVLM—a two-stage self-supervised large vision-language model that eliminates the need for manual annotations. It achieves fine-grained semantic decomposition and controllable generation via multi-granularity vision-language alignment, hierarchical captioning, and an internal compositional planning mechanism. We further introduce a semantic consistency loss and a self-refinement strategy to ensure cross-modal semantic fidelity and structural coherence. Extensive experiments on multiple benchmarks demonstrate that Hi-SSLVLM significantly outperforms state-of-the-art methods. Both quantitative metrics and human evaluations confirm its superiority in prompt fidelity, compositional accuracy, and aesthetic quality of generated images.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

This paper introduces Hierarchical Self-Supervised LVLM (Hi-SSLVLM), a novel generative model designed to significantly advance text-to-image synthesis, particularly for complex and compositionally challenging prompts. Traditional methods often grapple with the high cost of meticulously curated paired image-text datasets and struggle with precise control over fine-grained visual attributes and intricate spatial relationships. Our Hi-SSLVLM addresses these limitations through a unique two-stage self-supervised learning strategy. The first stage, Multi-Granularity Visual-Language Grounding, enables the Large Vision-Language Model (LVLM) backbone to autonomously generate and align hierarchical captions (global and local) to images, cultivating a deep internal semantic understanding without reliance on extensive human annotation. The second stage, Self-Refinement and Guided Image Generation, leverages this acquired knowledge by an Internal Compositional Planning (ICP) mechanism, where the LVLM first formulates detailed textual sub-prompts to guide the image generation process, complemented by a novel Semantic Consistency Loss for precise output alignment. Comprehensive experiments against leading baselines, including Janus-Pro-1B, Stable Diffusion XL 1.0, DeepFloyd IF v1.0, and ControlNet-XL, on multi-dimensional benchmarks such as Gemini-2.0-Flash and InternVL3-78B, demonstrate Hi-SSLVLM's superior performance across all fine-grained metrics. An in-depth ablation study confirms the critical role of each proposed component. Furthermore, human evaluations corroborate our quantitative findings, highlighting Hi-SSLVLM's enhanced fidelity to prompt, compositional accuracy, and overall aesthetic quality, marking a significant step towards more controllable and semantically consistent open-ended text-to-image generation.

Problem

• Advancing text-to-image synthesis for complex prompts
• Reducing reliance on costly curated image-text datasets
• Improving control over fine-grained visual attributes

Innovation

• Hierarchical self-supervised learning for LVLM
• Multi-granularity visual-language grounding without annotation
• Internal compositional planning for precise generation