Synthesizing 3D Abstractions by Inverting Procedural Buildings with Transformers

2025-01-28

AI Summary

This paper addresses the problem of reconstructing structured 3D models and completing missing regions from single-frame architectural point clouds. We propose an end-to-end differentiable “point cloud → program” inverse mapping method. Our core contribution is the first integration of procedural architectural modeling—specifically L-systems and CGA (Computer Generated Architecture)—as a structural prior into point cloud understanding, leveraging a Transformer architecture to learn a mapping from raw point clouds to executable, editable procedural descriptions. The method jointly supports geometric reconstruction, structural fidelity, symmetry constraints, and structurally consistent inpainting of occluded or missing regions. Evaluated on synthetic architectural point cloud datasets, our approach generates models exhibiting both high geometric accuracy and strong adherence to architectural regularities. The output programs are directly usable for real-time rendering, interactive editing, and downstream applications in digital twins and generative design.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

We generate abstractions of buildings, reflecting the essential aspects of their geometry and structure, by learning to invert procedural models. We first build a dataset of abstract procedural building models paired with simulated point clouds and then learn the inverse mapping through a transformer. Given a point cloud, the trained transformer then infers the corresponding abstracted building in terms of a programmatic language description. This approach leverages expressive procedural models developed for gaming and animation, and thereby retains desirable properties such as efficient rendering of the inferred abstractions and strong priors for regularity and symmetry. Our approach achieves good reconstruction accuracy in terms of geometry and structure, as well as structurally consistent inpainting.

Problem

• 3D Point Cloud
• Reconstruction
• Symmetry Restoration

Innovation

• Transformer-based abstraction
• Point cloud reconstruction
• Symmetry preservation