BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds

2026-02-27

AI Summary

This work addresses the highly under-constrained problem of reconstructing structured 3D buildings from sparse and irregularly distributed point clouds, such as those from airborne LiDAR or Structure-from-Motion (SfM) data. The authors propose a novel generative framework featuring a Loosely Cascaded Diffusion Transformer architecture, which decouples the task into two stages: first, a conditional latent diffusion model recovers the underlying geometric distribution of the input point cloud; second, a decoder-only Transformer autoregressively generates compact, structured building meshes. By effectively integrating diffusion-based generative modeling with autoregressive sequence generation, the method significantly outperforms existing approaches in both architectural abstraction quality and point cloud completion, producing surfaces that are more geometrically accurate and uniformly distributed.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce BuildAnyPoint, a novel generative framework for structured 3D building reconstruction from point clouds with diverse distributions, such as those captured by airborne LiDAR and Structure-from-Motion. To recover artist-created building abstraction in this highly underconstrained setting, we capitalize on the role of explicit 3D generative priors in autoregressive mesh generation. Specifically, we design a Loosely Cascaded Diffusion Transformer (Loca-DiT) that initially recovers the underlying distribution from noisy or sparse points, followed by autoregressively encapsulating them into compact meshes. We first formulate distribution recovery as a conditional generation task by training latent diffusion models conditioned on input point clouds, and then tailor a decoder-only transformer for conditional autoregressive mesh generation based on the recovered point clouds. Our method delivers substantial qualitative and quantitative improvements over prior building abstraction methods. Furthermore, the effectiveness of our approach is evidenced by the strong performance of its recovered point clouds on building point cloud completion benchmarks, which exhibit improved surface accuracy and distribution uniformity.

Problem

• 3D building reconstruction
• point cloud
• structured abstraction
• generative modeling
• distribution recovery

Innovation

• structured 3D reconstruction
• generative prior
• diffusion transformer
• autoregressive mesh generation
• point cloud abstraction