unPIC: A Geometric Multiview Prior for Image to 3D Synthesis

2024-12-13

AI Summary

This work addresses the generative problem of multi-view-consistent 3D reconstruction from a single 2D image. We propose a hierarchical probabilistic diffusion framework: first, a point-map-driven geometric prior module predicts a structured 3D point cloud; second, high-fidelity, geometrically consistent novel views are decoded conditioned on this point cloud. Our key contributions are (i) the first introduction of point-map representations as a multi-view geometric prior, enabling generalization to arbitrary input images; and (ii) a modular design that decouples geometric modeling from appearance synthesis, significantly improving cross-object transferability. Evaluated on real-world datasets—including ObjaverseXL and Google Scanned Objects—our method surpasses state-of-the-art approaches such as CAT3D and Free3D in both novel-view synthesis quality (FID, LPIPS) and 3D consistency (Chamfer distance), achieving new performance benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce a hierarchical probabilistic approach to go from a 2D image to multiview 3D: a diffusion"prior"predicts the unseen 3D geometry, which then conditions a diffusion"decoder"to generate novel views of the subject. We use a pointmap-based geometric representation to coordinate the generation of multiple target views simultaneously. We construct a predictable distribution of geometric features per target view to enable learnability across examples, and generalization to arbitrary inputs images. Our modular, geometry-driven approach to novel-view synthesis (called"unPIC") beats competing baselines such as CAT3D, EscherNet, Free3D, and One-2-3-45 on held-out objects from ObjaverseXL, as well as unseen real-world objects from Google Scanned Objects, Amazon Berkeley Objects, and the Digital Twin Catalog.

Problem

• Predict 3D geometry from a single 2D image
• Generate novel views using geometric representation
• Improve generalization for arbitrary input images

Innovation

• Hierarchical probabilistic approach for 2D to 3D
• Pointmap-based geometric multiview representation
• Predictable geometric feature distribution per view