Training-Free Generation of Diverse and High-Fidelity Images via Prompt Semantic Space Optimization

2025-11-24

AI Summary

Text-to-image diffusion models often suffer from semantic distribution collapse, leading to insufficient generation diversity and hindering creative applications. To address this, we propose Token-Prompt Embedding Space Optimization—a model-agnostic method that requires no fine-tuning. By jointly imposing learnable token-level and prompt-level constraints in the text embedding space, it actively explores low-frequency semantic regions to simultaneously enhance diversity and preserve generation quality. The approach is plug-and-play and compatible with diverse diffusion architectures. On MS-COCO, it significantly improves the diversity metric (LPIPS) from 1.10 to 4.18 while maintaining high fidelity and strong text–image alignment. This effectively mitigates mode collapse without compromising semantic consistency or visual quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity models tend to generate repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A primary cause is that generation often collapses toward a strong mode in the learned distribution. Existing attempts to improve diversity, such as noise resampling, prompt rewriting, or steering-based guidance, often still collapse to dominant modes or introduce distortions that degrade image quality. In light of this, we propose Token-Prompt embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency of the model to repeatedly generate samples from strong modes of the learned distribution. At the same time, the prompt-level space provides a global semantic constraint that regulates distribution shifts, preventing quality degradation while maintaining high fidelity. Extensive experiments on MS-COCO and three diffusion backbones show that TPSO significantly enhances generative diversity, improving baseline performance from 1.10 to 4.18 points, without sacrificing image quality. Code will be released upon acceptance.

Problem

• Addressing low diversity in text-to-image diffusion models
• Reducing repetitive outputs from dominant distribution modes
• Enhancing image variety without compromising quality fidelity

Innovation

• Training-free optimization of token embedding space
• Explores underrepresented regions to reduce repetition
• Maintains high fidelity with prompt-level semantic constraints