ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstruction

2025-11-10

AI Summary

This work addresses the end-to-end reconstruction of complex 3D assets from a single RGB image. We propose a neural procedural graph generation framework that represents 3D structure via differentiable procedural graphs, introduces an edge-based tokenization strategy, leverages Transformers to model structural sequence priors, and—crucially—incorporates Monte Carlo Tree Search (MCTS) for guided sampling, significantly improving image-to-3D alignment accuracy. Unlike prior approaches, our method requires neither category-specific priors nor multi-view supervision, enabling direct generation of decodable and editable 3D assets from monocular images. Evaluated on diverse complex objects—including cacti, trees, and bridges—our approach outperforms existing generative 3D methods and domain-specific modeling techniques in both fidelity and generalizability. Notably, it demonstrates strong generalization to real-world images while preserving fine-grained geometric and topological structure.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

We introduce ProcGen3D, a new approach for 3D content creation by generating procedural graph abstractions of 3D objects, which can then be decoded into rich, complex 3D assets. Inspired by the prevalent use of procedural generators in production 3D applications, we propose a sequentialized, graph-based procedural graph representation for 3D assets. We use this to learn to approximate the landscape of a procedural generator for image-based 3D reconstruction. We employ edge-based tokenization to encode the procedural graphs, and train a transformer prior to predict the next token conditioned on an input RGB image. Crucially, to enable better alignment of our generated outputs to an input image, we incorporate Monte Carlo Tree Search (MCTS) guided sampling into our generation process, steering output procedural graphs towards more image-faithful reconstructions. Our approach is applicable across a variety of objects that can be synthesized with procedural generators. Extensive experiments on cacti, trees, and bridges show that our neural procedural graph generation outperforms both state-of-the-art generative 3D methods and domain-specific modeling techniques. Furthermore, this enables improved generalization on real-world input images, despite training only on synthetic data.

Problem

• Learning neural procedural graph representations for 3D reconstruction
• Generating procedural graph abstractions from input RGB images
• Improving image-faithful 3D reconstruction using guided sampling

Innovation

• Generates procedural graph abstractions for 3D reconstruction
• Uses transformer with edge tokenization for sequential prediction
• Incorporates MCTS guided sampling for image-faithful outputs