The P$^3$ dataset: Pixels, Points and Polygons for Multimodal Building Vectorization

2025-05-21

AI Summary

Existing building vectorization methods heavily rely on image modalities and suffer from insufficient geometric accuracy. To address this, we introduce the first intercontinental, multimodal building vectorization benchmark, integrating 25 cm GSD aerial imagery, decimeter-level LiDAR point clouds (>1 billion points), and high-precision ground-truth vector footprints. We conduct the first systematic evaluation of LiDAR’s robustness for building vectorization and propose novel multimodal alignment strategies, joint point cloud–image encoding, and both end-to-end and two-stage prediction frameworks based on Transformers and CNNs. We publicly release the dataset, training code, and weights of three state-of-the-art models. Experiments demonstrate that image–LiDAR fusion substantially outperforms unimodal approaches, improving F1 and IoU by 8–12% on average, while significantly enhancing polygonal geometric integrity and topological correctness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present the P$^3$ dataset, a large-scale multimodal benchmark for building vectorization, constructed from aerial LiDAR point clouds, high-resolution aerial imagery, and vectorized 2D building outlines, collected across three continents. The dataset contains over 10 billion LiDAR points with decimeter-level accuracy and RGB images at a ground sampling distance of 25 centimeter. While many existing datasets primarily focus on the image modality, P$^3$ offers a complementary perspective by also incorporating dense 3D information. We demonstrate that LiDAR point clouds serve as a robust modality for predicting building polygons, both in hybrid and end-to-end learning frameworks. Moreover, fusing aerial LiDAR and imagery further improves accuracy and geometric quality of predicted polygons. The P$^3$ dataset is publicly available, along with code and pretrained weights of three state-of-the-art models for building polygon prediction at https://github.com/raphaelsulzer/PixelsPointsPolygons .

Problem

• Multimodal building vectorization using LiDAR and imagery
• Improving polygon prediction accuracy with 3D data fusion
• Large-scale benchmark for hybrid and end-to-end learning

Innovation

• Multimodal dataset with LiDAR, imagery, and polygons
• Hybrid and end-to-end learning frameworks for prediction
• Fusion of LiDAR and imagery improves polygon accuracy