SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting

2026-04-21

AI Summary

This work addresses the challenge of generating geometrically consistent, high-fidelity 3D Gaussian face models in real time from sparse, depth-ambiguous 2D sketches lacking high-frequency details. The authors propose an end-to-end, coarse-to-fine feedforward framework that integrates a Transformer-based UV feature map predictor with a 3D UV enhancement module. By introducing UV Mask Fusion and inter-layer feature fusion strategies, the method achieves, for the first time, sketch-driven real-time generation and free-view editing of 3D Gaussian faces. The entire pipeline produces high-quality, editable models in a single forward pass, outperforming existing approaches in both reconstruction fidelity and editing flexibility.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

3D Gaussian representations have emerged as a powerful paradigm for digital head modeling, achieving photorealistic quality with real-time rendering. However, intuitive and interactive creation or editing of 3D Gaussian head models remains challenging. Although 2D sketches provide an ideal interaction modality for fast, intuitive conceptual design, they are sparse, depth-ambiguous, and lack high-frequency appearance cues, making it difficult to infer dense, geometrically consistent 3D Gaussian structures from strokes - especially under real-time constraints. To address these challenges, we propose SketchFaceGS, the first sketch-driven framework for real-time generation and editing of photorealistic 3D Gaussian head models from 2D sketches. Our method uses a feed-forward, coarse-to-fine architecture. A Transformer-based UV feature-prediction module first reconstructs a coarse but geometrically consistent UV feature map from the input sketch, and then a 3D UV feature enhancement module refines it with high-frequency, photorealistic detail to produce a high-fidelity 3D head. For editing, we introduce a UV Mask Fusion technique combined with a layer-by-layer feature-fusion strategy, enabling precise, real-time, free-viewpoint modifications. Extensive experiments show that SketchFaceGS outperforms existing methods in both generation fidelity and editing flexibility, producing high-quality, editable 3D heads from sketches in a single forward pass.

Problem

• 3D Gaussian Splatting
• sketch-based modeling
• real-time editing
• face generation
• depth ambiguity

Innovation

• Gaussian Splatting
• Sketch-Based Modeling
• Real-Time 3D Editing
• UV Feature Prediction
• Transformer Architecture