PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction

2024-04-16

arXiv.org

AI Summary

This work addresses the challenge of simultaneously achieving shape interpretability, semantic editability, and memory efficiency in single-image 3D reconstruction. Methodologically, we propose a differentiable procedural modeling framework: (i) we introduce the first automatic translation of Blender Geometry Nodes—defining procedural 3D models—into differentiable PyTorch code, enabling joint gradient-based optimization over both discrete topological structures and continuous parameters; (ii) we integrate genetic algorithms for topology search and incorporate procedural Gaussian splatting for efficient, high-fidelity rendering. Our contributions include state-of-the-art, semantically aligned 3D reconstructions on ScanNet, with significantly improved shape editability and parameter interpretability—breaking away from conventional CAD model retrieval paradigms—while maintaining low memory overhead. The framework enables intuitive, semantics-aware editing via interpretable procedural parameters and scales efficiently to complex scenes without compromising reconstruction fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We propose PyTorchGeoNodes, a differentiable module for reconstructing 3D objects and their parameters from images using interpretable shape programs. Unlike traditional CAD model retrieval, shape programs allow reasoning about semantic parameters, editing, and a low memory footprint. Despite their potential, shape programs for 3D scene understanding have been largely overlooked. Our key contribution is enabling gradient-based optimization by parsing shape programs, or more precisely procedural models designed in Blender, into efficient PyTorch code. While there are many possible applications of our PyTochGeoNodes, we show that a combination of PyTorchGeoNodes with genetic algorithm is a method of choice to optimize both discrete and continuous shape program parameters for 3D reconstruction and understanding of 3D object parameters. Our modular framework can be further integrated with other reconstruction algorithms, and we demonstrate one such integration to enable procedural Gaussian splatting. Our experiments on the ScanNet dataset show that our method achieves accurate reconstructions while enabling, until now, unseen level of 3D scene understanding.

Problem

• Enabling differentiable 3D shape reconstruction from images
• Optimizing discrete and continuous shape program parameters
• Integrating interpretable shape programs with reconstruction algorithms

Innovation

• Differentiable shape programs for 3D reconstruction
• Gradient-based optimization of procedural models
• Integration with genetic algorithm for parameter optimization