SPF-Portrait: Towards Pure Portrait Customization with Semantic Pollution-Free Fine-tuning

2025-04-01

AI Summary

Existing text-to-image (T2I) models suffer from semantic contamination when fine-tuned on portrait datasets, degrading their original generalization capability and impeding incremental learning. To address this, we propose a contamination-free paradigm for pure portrait customization. Our method introduces a dual-path contrastive fine-tuning framework that decouples semantic and appearance learning; incorporates a semantic-aware fine-grained control map to enforce cross-modal spatial alignment; and designs a response enhancement mechanism to mitigate text–image supervision mismatch. Evaluated on multiple portrait customization benchmarks, our approach achieves state-of-the-art performance, significantly suppresses semantic drift, fully preserves the pretrained model’s general-purpose capabilities, and enables sustainable, scalable incremental customization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While fine-tuning pre-trained Text-to-Image (T2I) models on portrait datasets enables attribute customization, existing methods suffer from Semantic Pollution that compromises the original model's behavior and prevents incremental learning. To address this, we propose SPF-Portrait, a pioneering work to purely understand customized semantics while eliminating semantic pollution in text-driven portrait customization. In our SPF-Portrait, we propose a dual-path pipeline that introduces the original model as a reference for the conventional fine-tuning path. Through contrastive learning, we ensure adaptation to target attributes and purposefully align other unrelated attributes with the original portrait. We introduce a novel Semantic-Aware Fine Control Map, which represents the precise response regions of the target semantics, to spatially guide the alignment process between the contrastive paths. This alignment process not only effectively preserves the performance of the original model but also avoids over-alignment. Furthermore, we propose a novel response enhancement mechanism to reinforce the performance of target attributes, while mitigating representation discrepancy inherent in direct cross-modal supervision. Extensive experiments demonstrate that SPF-Portrait achieves state-of-the-art performance.

Problem

• Eliminates semantic pollution in portrait customization
• Preserves original model performance during fine-tuning
• Enhances target attribute representation without over-alignment

Innovation

• Dual-path pipeline with original model reference
• Semantic-Aware Fine Control Map for alignment
• Response enhancement mechanism for target attributes