Stage-Aware Adaptation and Distribution Calibration for Subject-Driven Personalized Text-to-Image Generation

2026-07-08

AI Summary

This work addresses the challenge of simultaneously achieving identity consistency, textual alignment, and generation diversity in few-shot personalized text-to-image synthesis. To this end, the authors propose a synergistic framework comprising Stage-aware Low-Rank Adaptation (SPaRa) and Distribution-Calibrated Candidate Selection (DCAL). SPaRa introduces timestep-dependent low-rank adaptation during training to dynamically optimize personalized representations across denoising stages, while DCAL enhances inference by selecting diverse yet semantically coherent candidates through feature-space distribution calibration—overcoming limitations of uniform adaptation strength and identity-similarity-only selection. Evaluated under the SDXL and DreamBooth 30-subject protocol, the method significantly improves 1-LPIPS, CLIP-I, DINO-I, and CLIP-T metrics, effectively revealing and balancing the inherent trade-offs among the three objectives.

Abstract

Subject-driven personalized text-to-image generation requires a pretrained diffusion model to acquire a specific subject from a few reference images while preserving subject identity, following novel text prompts, and maintaining sample diversity. Existing optimization-based methods instantiate subject adaptation through full fine-tuning, textual embedding optimization, or low-rank parameter updates; PaRa further constrains personalization from the perspective of parameter rank reduction. However, a uniform low-rank constraint or a uniform adapter strength cannot explicitly distinguish the capacity requirements of different denoising stages. Moreover, inference-time candidate selection driven mainly by identity similarity may compress the selected samples in the visual representation space. We decompose the problem into two complementary components: SPaRa denotes training-side stage-aware low-rank adaptation, DCAL denotes inference-side distribution-calibrated candidate selection, and SPaRa-DCAL denotes the combined framework. Theoretical analysis shows that timestep-dependent scaling controls the effective perturbation magnitude of a low-rank adapter, while identity-biased candidate selection restricts the radius of selected features around the reference center under explicit conditions. Auditable experiments under the SDXL and DreamBooth 30-subject protocol show that DCAL improves 1-LPIPS, CLIP-I, DINO-I, and CLIP-T on a fixed LoRA candidate pool, while revealing a clear trade-off with CLIP/DINO pairwise diversity and pairwise LPIPS. These results indicate that personalized generation should be evaluated through identity consistency, text alignment, and representation diversity rather than identity metrics alone.

Problem

• personalized text-to-image generation
• subject identity
• sample diversity
• diffusion model
• few-shot adaptation

Innovation

• stage-aware adaptation
• distribution calibration
• low-rank adaptation
• personalized text-to-image generation
• candidate selection