SuperNeRF-GAN: A Universal 3D-Consistent Super-Resolution Framework for Efficient and Enhanced 3D-Aware Image Synthesis

2025-01-12

AI Summary

NeRF-based methods face two key challenges in high-resolution 3D synthesis: prohibitive computational cost and 3D inconsistency after super-resolution. To address these, this paper proposes the first general, plug-and-play 3D-consistent super-resolution framework compatible with arbitrary NeRF generators (e.g., tri-plane representations). Our core innovation is a depth-guided rendering pipeline: we construct boundary-corrected multi-depth maps, introduce normal-constrained depth super-resolution, and leverage the super-resolved depth to guide NeRF re-rendering. This design strictly enforces both geometric and view consistency while enhancing resolution. Experiments demonstrate a 2.3× speedup in inference time and significant PSNR/SSIM improvements over state-of-the-art methods. Ablation studies validate the effectiveness and necessity of each component.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Consent frameworks and practices on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Neural volume rendering techniques, such as NeRF, have revolutionized 3D-aware image synthesis by enabling the generation of images of a single scene or object from various camera poses. However, the high computational cost of NeRF presents challenges for synthesizing high-resolution (HR) images. Most existing methods address this issue by leveraging 2D super-resolution, which compromise 3D-consistency. Other methods propose radiance manifolds or two-stage generation to achieve 3D-consistent HR synthesis, yet they are limited to specific synthesis tasks, reducing their universality. To tackle these challenges, we propose SuperNeRF-GAN, a universal framework for 3D-consistent super-resolution. A key highlight of SuperNeRF-GAN is its seamless integration with NeRF-based 3D-aware image synthesis methods and it can simultaneously enhance the resolution of generated images while preserving 3D-consistency and reducing computational cost. Specifically, given a pre-trained generator capable of producing a NeRF representation such as tri-plane, we first perform volume rendering to obtain a low-resolution image with corresponding depth and normal map. Then, we employ a NeRF Super-Resolution module which learns a network to obtain a high-resolution NeRF. Next, we propose a novel Depth-Guided Rendering process which contains three simple yet effective steps, including the construction of a boundary-correct multi-depth map through depth aggregation, a normal-guided depth super-resolution and a depth-guided NeRF rendering. Experimental results demonstrate the superior efficiency, 3D-consistency, and quality of our approach. Additionally, ablation studies confirm the effectiveness of our proposed components.

Problem

• Neural Radiance Fields
• Super-Resolution
• 3D Image Generation

Innovation

• SuperNeRF-GAN
• 3D Image Synthesis
• High-resolution Enhancement