Domain Generalizable Portrait Style Transfer

2025-07-06

AI Summary

This paper addresses cross-domain generalization in high-fidelity portrait style transfer, targeting semantically aligned stylization across multiple regions—including hair, eyes, eyelashes, skin, lips, and background. Methodologically, it introduces semantic and style adapters to establish dense semantic correspondences between input and reference images; incorporates AdaIN-Wavelet transformation in the latent space to jointly fuse high- and low-frequency features for fine-grained style control while preserving structural content; and integrates a dual-condition diffusion model based on ControlNet to enable semantics-guided, high-quality generation. Experiments demonstrate superior performance across diverse domains, significantly improving regional semantic consistency and perceptual image quality, while exhibiting strong generalization capability. The code and pretrained models are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Application Domains: Humanities & Computational Social Science

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper presents a portrait style transfer method that generalizes well to various different domains while enabling high-quality semantic-aligned stylization on regions including hair, eyes, eyelashes, skins, lips, and background. To this end, we propose to establish dense semantic correspondence between the given input and reference portraits based on a pre-trained model and a semantic adapter, with which we obtain a warped reference semantically aligned with the input. To ensure effective yet controllable style transfer, we devise an AdaIN-Wavelet transform to balance content preservation and stylization by blending low-frequency information of the warped reference with high-frequency information of the input in the latent space. A style adapter is also designed to provide style guidance from the warped reference. With the stylized latent from AdaIN-Wavelet transform, we employ a dual-conditional diffusion model that integrates a ControlNet recording high-frequency information and the style guidance to generate the final result. Extensive experiments demonstrate the superiority of our method. Our code and trained model are available at https://github.com/wangxb29/DGPST.

Problem

• Generalize portrait style transfer across diverse domains
• Achieve semantic-aligned stylization for detailed facial regions
• Balance content preservation and stylization controllably

Innovation

• Dense semantic correspondence via pre-trained model
• AdaIN-Wavelet transform for balanced stylization
• Dual-conditional diffusion model with ControlNet