Styleclone: Face Stylization with Diffusion Based Data Augmentation

2025-08-23

AI Summary

To address data scarcity, limited diversity, and content distortion in few-shot facial style transfer, this paper proposes an efficient style data augmentation framework integrating diffusion models with textual inversion. First, textual inversion learns compact, disentangled embeddings of target styles from minimal reference images; then, a diffusion model—guided by these embeddings—generates diverse, high-fidelity stylized face images. The augmented dataset trains a lightweight image-to-image translation network. Evaluated with only 2–5 style reference images, our method significantly improves stylization quality and preserves identity/structural fidelity across diverse artistic styles, outperforming existing few-shot approaches. Inference is 20–50× faster than native diffusion models, achieving an optimal trade-off between quality and efficiency. Moreover, this work introduces the first systematic evaluation of data augmentation strategies specifically designed for style transfer, establishing a new benchmark for augmentation-driven few-shot stylization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We present StyleClone, a method for training image-to-image translation networks to stylize faces in a specific style, even with limited style images. Our approach leverages textual inversion and diffusion-based guided image generation to augment small style datasets. By systematically generating diverse style samples guided by both the original style images and real face images, we significantly enhance the diversity of the style dataset. Using this augmented dataset, we train fast image-to-image translation networks that outperform diffusion-based methods in speed and quality. Experiments on multiple styles demonstrate that our method improves stylization quality, better preserves source image content, and significantly accelerates inference. Additionally, we provide a systematic evaluation of the augmentation techniques and their impact on stylization performance.

Problem

• Augmenting limited style datasets for face stylization
• Training fast image-to-image translation networks
• Improving stylization quality while preserving content

Innovation

• Diffusion-based guided image generation
• Textual inversion for data augmentation
• Fast image-to-image translation networks