Training-free Stylized Text-to-Image Generation with Fast Inference

2025-05-25

AI Summary

Existing diffusion-based stylized image generation methods rely on computationally expensive text inversion or fine-tuning, suffering from poor generalization and high training overhead. Method: We propose the first training-free, fine-tuning-free style transfer paradigm for latent diffusion models (LDMs). Our approach extracts channel-wise statistical features from a single reference style image and introduces a self-attention norm mixing mechanism to dynamically align content structure with style characteristics—all within the fixed pre-trained LDM. Leveraging the self-consistency prior of latent consistency models, it requires no parameter updates. Contribution/Results: Our method achieves state-of-the-art performance across multiple benchmarks, delivering high-fidelity style transfer and photorealistic image generation while maintaining millisecond-level inference speed—demonstrating unprecedented efficiency, generalizability, and fidelity without any optimization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Although diffusion models exhibit impressive generative capabilities, existing methods for stylized image generation based on these models often require textual inversion or fine-tuning with style images, which is time-consuming and limits the practical applicability of large-scale diffusion models. To address these challenges, we propose a novel stylized image generation method leveraging a pre-trained large-scale diffusion model without requiring fine-tuning or any additional optimization, termed as OmniPainter. Specifically, we exploit the self-consistency property of latent consistency models to extract the representative style statistics from reference style images to guide the stylization process. Additionally, we then introduce the norm mixture of self-attention, which enables the model to query the most relevant style patterns from these statistics for the intermediate output content features. This mechanism also ensures that the stylized results align closely with the distribution of the reference style images. Our qualitative and quantitative experimental results demonstrate that the proposed method outperforms state-of-the-art approaches.

Problem

• Enables training-free stylized image generation without fine-tuning
• Extracts style statistics from reference images for guidance
• Ensures stylized results match reference style distribution

Innovation

• Training-free stylized image generation method
• Leverages pre-trained diffusion model without fine-tuning
• Norm mixture of self-attention for style alignment