WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting

2025-11-11

AI Summary

Existing single-image 3D GAN inversion methods rely on generator priors to inpaint occluded regions in novel-view synthesis, but low-bitrate latent codes often cause information loss, resulting in distorted occlusions and multi-view inconsistency. To address this, we propose WarpGAN—the first framework that jointly integrates depth-guided geometric warping with the SVINet-style refinement network. Leveraging symmetric priors and cross-view correspondence constraints, WarpGAN co-optimizes the 3D inversion encoder, depth estimator, warping module, and refinement network. This enables high-fidelity reconstruction of visible regions while significantly improving realism and multi-view consistency in occluded areas. Evaluated on multiple benchmarks, WarpGAN outperforms state-of-the-art methods, achieving substantial gains in occlusion quality and geometric consistency.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

3D GAN inversion projects a single image into the latent space of a pre-trained 3D GAN to achieve single-shot novel view synthesis, which requires visible regions with high fidelity and occluded regions with realism and multi-view consistency. However, existing methods focus on the reconstruction of visible regions, while the generation of occluded regions relies only on the generative prior of 3D GAN. As a result, the generated occluded regions often exhibit poor quality due to the information loss caused by the low bit-rate latent code. To address this, we introduce the warping-and-inpainting strategy to incorporate image inpainting into 3D GAN inversion and propose a novel 3D GAN inversion method, WarpGAN. Specifically, we first employ a 3D GAN inversion encoder to project the single-view image into a latent code that serves as the input to 3D GAN. Then, we perform warping to a novel view using the depth map generated by 3D GAN. Finally, we develop a novel SVINet, which leverages the symmetry prior and multi-view image correspondence w.r.t. the same latent code to perform inpainting of occluded regions in the warped image. Quantitative and qualitative experiments demonstrate that our method consistently outperforms several state-of-the-art methods.

Problem

• Improving occluded region generation in 3D GAN inversion
• Addressing information loss from low bit-rate latent codes
• Enhancing multi-view consistency for novel view synthesis

Innovation

• Warping-and-inpainting strategy enhances 3D GAN inversion
• SVINet uses symmetry prior for occluded region inpainting
• Depth map warping enables novel view synthesis