A Controllable 3D Deepfake Generation Framework with Gaussian Splatting

2025-09-15

AI Summary

Traditional 2D deepfake methods suffer from poor geometric consistency and limited novel-view generalization, hindering controllable 3D face swapping and reenactment. To address this, we propose the first controllable deepfake framework based on 3D Gaussian splatting: it models facial geometry and expressions via head-separate dynamic Gaussians, integrated with a parametric head prior and multi-view joint optimization. We further introduce a 2D diffusion-guided rendering refinement module and a pose-robust inpainting module to enhance visual consistency under extreme viewpoints. This work is the first to systematically apply 3D Gaussian splatting to deepfaking. On benchmarks including NeRSemble, our method achieves identity preservation comparable to state-of-the-art 2D approaches, while significantly improving multi-view consistency, 3D geometric plausibility, and natural background integration—demonstrating superior 3D controllability and scene adaptability.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Responsible Web: Consent frameworks and practices on the web
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We propose a novel 3D deepfake generation framework based on 3D Gaussian Splatting that enables realistic, identity-preserving face swapping and reenactment in a fully controllable 3D space. Compared to conventional 2D deepfake approaches that suffer from geometric inconsistencies and limited generalization to novel view, our method combines a parametric head model with dynamic Gaussian representations to support multi-view consistent rendering, precise expression control, and seamless background integration. To address editing challenges in point-based representations, we explicitly separate the head and background Gaussians and use pre-trained 2D guidance to optimize the facial region across views. We further introduce a repair module to enhance visual consistency under extreme poses and expressions. Experiments on NeRSemble and additional evaluation videos demonstrate that our method achieves comparable performance to state-of-the-art 2D approaches in identity preservation, as well as pose and expression consistency, while significantly outperforming them in multi-view rendering quality and 3D consistency. Our approach bridges the gap between 3D modeling and deepfake synthesis, enabling new directions for scene-aware, controllable, and immersive visual forgeries, revealing the threat that emerging 3D Gaussian Splatting technique could be used for manipulation attacks.

Problem

• Generating 3D deepfakes with controllable face swapping
• Achieving multi-view consistent rendering in deepfakes
• Enhancing visual consistency under extreme poses

Innovation

• 3D Gaussian Splatting deepfake framework
• Parametric head model with dynamic Gaussians
• View-consistent rendering with repair module