Style-Friendly SNR Sampler for Style-Driven Generation

2024-11-22

arXiv.org

AI Summary

Existing text-to-image diffusion models generate high-quality images but struggle to effectively learn user-provided personalized styles—especially when guided by reference images rather than textual descriptions—due to fixed noise schedules and signal-to-noise ratio (SNR) distributions misaligned with the style-specific distribution in high-noise regimes. This work proposes RefStyle: first, we empirically observe that style representations are most discriminative at high noise levels; second, we introduce a dynamic SNR offset strategy to concentrate sampling in this critical regime; third, we integrate fine-grained noise schedule optimization, style-aware SNR reweighting, and a lightweight reference-guided fine-tuning paradigm. Experiments demonstrate that RefStyle significantly enhances modeling of novel, textually ambiguous styles while preserving content fidelity, enabling high-quality, high-fidelity personalized style transfer.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Recent text-to-image diffusion models generate high-quality images but struggle to learn new, personalized styles, which limits the creation of unique style templates. In style-driven generation, users typically supply reference images exemplifying the desired style, together with text prompts that specify desired stylistic attributes. Previous approaches popularly rely on fine-tuning, yet it often blindly utilizes objectives and noise level distributions from pre-training without adaptation. We discover that stylistic features predominantly emerge at higher noise levels, leading current fine-tuning methods to exhibit suboptimal style alignment. We propose the Style-friendly SNR sampler, which aggressively shifts the signal-to-noise ratio (SNR) distribution toward higher noise levels during fine-tuning to focus on noise levels where stylistic features emerge. This enhances models' ability to capture novel styles indicated by reference images and text prompts. We demonstrate improved generation of novel styles that cannot be adequately described solely with a text prompt, enabling the creation of new style templates for personalized content creation.

Problem

• Struggle to learn new personalized styles in text-to-image models
• Suboptimal style alignment due to improper noise level distribution
• Need for better style capture from reference images and text prompts

Innovation

• Style-friendly SNR sampler for fine-tuning
• Shifts SNR distribution to higher noise levels
• Enhances capture of novel styles from references