FacaDiffy: Inpainting Unseen Facade Parts Using Diffusion Models

2025-02-20

AI Summary

To address incompleteness in 2D conflict maps caused by occlusions in laser scanning, this paper proposes a facade unseen-region inpainting method based on a customized Stable Diffusion model to enable high-fidelity semantic 3D modeling. Our approach comprises three key contributions: (1) the first application of LoRA-finetuned Stable Diffusion for conflict map completion; (2) a deterministic ray-casting algorithm that generates high-quality synthetic ground truth; and (3) a scalable synthetic data pipeline integrating stochastic urban modeling, facade rendering, and automatic annotation. Evaluated on both real and synthetic datasets, our method achieves state-of-the-art performance in conflict map completion. When integrated into semantic 3D reconstruction, it improves opening detection accuracy by 22%. The source code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

High-detail semantic 3D building models are frequently utilized in robotics, geoinformatics, and computer vision. One key aspect of creating such models is employing 2D conflict maps that detect openings' locations in building facades. Yet, in reality, these maps are often incomplete due to obstacles encountered during laser scanning. To address this challenge, we introduce FacaDiffy, a novel method for inpainting unseen facade parts by completing conflict maps with a personalized Stable Diffusion model. Specifically, we first propose a deterministic ray analysis approach to derive 2D conflict maps from existing 3D building models and corresponding laser scanning point clouds. Furthermore, we facilitate the inpainting of unseen facade objects into these 2D conflict maps by leveraging the potential of personalizing a Stable Diffusion model. To complement the scarcity of real-world training data, we also develop a scalable pipeline to produce synthetic conflict maps using random city model generators and annotated facade images. Extensive experiments demonstrate that FacaDiffy achieves state-of-the-art performance in conflict map completion compared to various inpainting baselines and increases the detection rate by $22%$ when applying the completed conflict maps for high-definition 3D semantic building reconstruction. The code is be publicly available in the corresponding GitHub repository: https://github.com/ThomasFroech/InpaintingofUnseenFacadeObjects

Problem

• Inpainting incomplete facade conflict maps
• Personalized Stable Diffusion model usage
• Enhancing 3D semantic building reconstruction

Innovation

• Personalized Stable Diffusion model
• Deterministic ray analysis approach
• Synthetic conflict maps pipeline