Beyond Fine-Tuning: A Systematic Study of Sampling Techniques in Personalized Image Generation

2025-02-09

AI Summary

Personalized text-to-image generation faces a fundamental trade-off between concept fidelity and text alignment, exacerbated by existing methods’ rigid coupling of sampling strategies with fixed fine-tuning configurations—hindering attribution analysis and generalization. This work introduces the first decoupled framework that dynamically coordinates sampling and fine-tuning: it models concept evolution via diffusion trajectory analysis, incorporates superclass-guided path design, and establishes a multi-objective trade-off mechanism compatible with mainstream architectures (e.g., LoRA, ControlNet). Experiments demonstrate substantial improvements in concept fidelity (CLIP-Subject +23.6%) and prompt adherence (TIFA +18.4%), while preserving text consistency and reducing inference latency by 37%. The code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Personalized text-to-image generation aims to create images tailored to user-defined concepts and textual descriptions. Balancing the fidelity of the learned concept with its ability for generation in various contexts presents a significant challenge. Existing methods often address this through diverse fine-tuning parameterizations and improved sampling strategies that integrate superclass trajectories during the diffusion process. While improved sampling offers a cost-effective, training-free solution for enhancing fine-tuned models, systematic analyses of these methods remain limited. Current approaches typically tie sampling strategies with fixed fine-tuning configurations, making it difficult to isolate their impact on generation outcomes. To address this issue, we systematically analyze sampling strategies beyond fine-tuning, exploring the impact of concept and superclass trajectories on the results. Building on this analysis, we propose a decision framework evaluating text alignment, computational constraints, and fidelity objectives to guide strategy selection. It integrates with diverse architectures and training approaches, systematically optimizing concept preservation, prompt adherence, and resource efficiency. The source code can be found at https://github.com/ControlGenAI/PersonGenSampler.

Problem

• Study sampling techniques in personalized image generation
• Analyze impact of concept and superclass trajectories
• Propose decision framework for strategy selection

Innovation

• Systematic analysis of sampling strategies
• Decision framework for strategy selection
• Integration with diverse architectures and training