DreamMakeup: Face Makeup Customization using Latent Diffusion Models

2025-10-12

AI Summary

Existing virtual makeup methods suffer from training instability and limited customization flexibility. To address these issues, this paper proposes a training-free latent diffusion model framework. Our approach preserves facial structure and identity features via early-stopped DDIM inversion, enabling high-fidelity, fine-grained makeup editing conditioned on multimodal inputs—including reference images, RGB color values, and textual descriptions. We introduce a novel multi-condition guidance mechanism and integrate a text encoder with a large language model interface, ensuring high editing quality under low computational overhead. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art GAN- and diffusion-based approaches in identity preservation, color matching accuracy, text alignment, and customization freedom. To the best of our knowledge, this is the first diffusion-based virtual makeup framework achieving multimodal controllability, zero-shot adaptation, and high fidelity without any training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The exponential growth of the global makeup market has paralleled advancements in virtual makeup simulation technology. Despite the progress led by GANs, their application still encounters significant challenges, including training instability and limited customization capabilities. Addressing these challenges, we introduce DreamMakup - a novel training-free Diffusion model based Makeup Customization method, leveraging the inherent advantages of diffusion models for superior controllability and precise real-image editing. DreamMakeup employs early-stopped DDIM inversion to preserve the facial structure and identity while enabling extensive customization through various conditioning inputs such as reference images, specific RGB colors, and textual descriptions. Our model demonstrates notable improvements over existing GAN-based and recent diffusion-based frameworks - improved customization, color-matching capabilities, identity preservation and compatibility with textual descriptions or LLMs with affordable computational costs.

Problem

• Addressing GANs' training instability and limited customization capabilities
• Enabling precise makeup customization through diffusion-based real-image editing
• Preserving facial identity while allowing diverse conditioning inputs

Innovation

• Training-free diffusion model for makeup customization
• Early-stopped DDIM inversion preserves facial identity
• Multiple conditioning inputs enable extensive customization