CraftGraffiti: Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models

2025-08-28

AI Summary

Preserving facial identity features—such as fine-grained structures of eyes, nose, and mouth—remains challenging in extreme stylized graffiti art generation, especially under high contrast and strong abstraction, which often cause identity collapse. To address this, we propose an end-to-end text-guided “style-first, identity-second” generation framework. First, LoRA-finetuned diffusion models perform graffiti-style transfer. Second, we introduce an explicit identity-embedding-enhanced facial consistency self-attention mechanism, coupled with CLIP-guided pose prompting to dynamically preserve both pose and identity without requiring facial landmark annotations. Our method significantly outperforms existing approaches in facial feature fidelity, achieves state-of-the-art aesthetic scores and user preference ratings, and has been successfully validated in real-world artistic creation at the Cruilla Music Festival.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Preserving facial identity under extreme stylistic transformation remains a major challenge in generative art. In graffiti, a high-contrast, abstract medium, subtle distortions to the eyes, nose, or mouth can erase the subject's recognizability, undermining both personal and cultural authenticity. We present CraftGraffiti, an end-to-end text-guided graffiti generation framework designed with facial feature preservation as a primary objective. Given an input image and a style and pose descriptive prompt, CraftGraffiti first applies graffiti style transfer via LoRA-fine-tuned pretrained diffusion transformer, then enforces identity fidelity through a face-consistent self-attention mechanism that augments attention layers with explicit identity embeddings. Pose customization is achieved without keypoints, using CLIP-guided prompt extension to enable dynamic re-posing while retaining facial coherence. We formally justify and empirically validate the "style-first, identity-after" paradigm, showing it reduces attribute drift compared to the reverse order. Quantitative results demonstrate competitive facial feature consistency and state-of-the-art aesthetic and human preference scores, while qualitative analyses and a live deployment at the Cruilla Festival highlight the system's real-world creative impact. CraftGraffiti advances the goal of identity-respectful AI-assisted artistry, offering a principled approach for blending stylistic freedom with recognizability in creative AI applications.

Problem

• Preserving facial identity in extreme graffiti style transformations
• Preventing facial feature distortion during artistic style transfer
• Balancing stylistic freedom with facial recognizability in generative art

Innovation

• Facial-preserving diffusion models
• Face-consistent self-attention mechanism
• CLIP-guided prompt extension