Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging

2025-03-28

AI Summary

To address domain gaps and geometric ambiguities in high-fidelity 3D geometry reconstruction from a single RGB image, this paper proposes a three-stage framework bridging via surface normal maps. First, a noise-decoupled dual-stream CNN enables robust RGB-to-normal estimation. Second, a normal-guided latent diffusion model generates regularized, globally consistent normal fields. Third, an end-to-end differentiable rendering pipeline—trained on synthetic data—refines geometry via gradient-based optimization. Crucially, this work pioneers the use of normal maps as a stable, geometrically unambiguous intermediate representation, effectively mitigating fine-detail loss induced by RGB ambiguity. Extensive evaluations demonstrate significant improvements in fine-grained geometry reconstruction across multiple benchmarks, particularly excelling in recovering intricate carving textures, thin-walled structures, and subtle concave–convex details—surpassing state-of-the-art methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

With the growing demand for high-fidelity 3D models from 2D images, existing methods still face significant challenges in accurately reproducing fine-grained geometric details due to limitations in domain gaps and inherent ambiguities in RGB images. To address these issues, we propose Hi3DGen, a novel framework for generating high-fidelity 3D geometry from images via normal bridging. Hi3DGen consists of three key components: (1) an image-to-normal estimator that decouples the low-high frequency image pattern with noise injection and dual-stream training to achieve generalizable, stable, and sharp estimation; (2) a normal-to-geometry learning approach that uses normal-regularized latent diffusion learning to enhance 3D geometry generation fidelity; and (3) a 3D data synthesis pipeline that constructs a high-quality dataset to support training. Extensive experiments demonstrate the effectiveness and superiority of our framework in generating rich geometric details, outperforming state-of-the-art methods in terms of fidelity. Our work provides a new direction for high-fidelity 3D geometry generation from images by leveraging normal maps as an intermediate representation.

Problem

• Generating high-fidelity 3D models from 2D images
• Overcoming domain gaps and RGB ambiguities in 3D generation
• Enhancing geometric detail accuracy via normal bridging

Innovation

• Image-to-normal estimator with dual-stream training
• Normal-to-geometry learning via latent diffusion
• High-quality 3D data synthesis pipeline