Shell-Supervised Gaussian Splatting for Urban Real-to-Sim Reconstruction

2026-06-29

AI Summary

This work addresses the geometric instability and limited agent-interaction support in urban facade reconstruction from real-world imagery to simulation, caused by glass surfaces, reflections, and repetitive textures. To tackle this, the authors propose a Gaussian splatting framework that incorporates an external building facade structural shell as a lightweight geometric prior. During optimization, a mask-gated loss is applied only to regions both supported by the structural shell and visible in the input views, jointly constraining rendered depth, normals, and valid masks to balance photometric fidelity with geometric consistency. Experiments demonstrate that the proposed method significantly improves alignment between facade orientations and surface point clouds compared to image-only, monocular, and surface-aware baselines, while maintaining high-quality novel view synthesis.

Abstract

Real-to-sim reconstruction for embodied AI requires geometry that is useful for collision reasoning, navigation, and agent-environment interaction, not only photorealistic novel-view synthesis. However, close-range urban facades are difficult for video-to-3D reconstruction: glass, reflections, repeated windows, and weak texture can produce visually plausible renderings with unstable surface geometry. We introduce shell-supervised Gaussian Splatting, a reconstruction-stage framework that uses an external facade structural shell as lightweight geometric supervision for video-driven Gaussian reconstruction. The method aligns an exterior shell to the video reconstruction frame, renders per-view depth, camera-space normal, and valid-mask maps, and applies these cues through mask-gated losses during Gaussian optimization. This design preserves RGB-driven appearance while regularizing only visible shell-supported facade regions. Experiments on anonymized close-range urban facade scenes show improved facade orientation and visible-surface point-cloud consistency over photo-only, monocular-cue, and surface-oriented Gaussian baselines, while maintaining comparable held-out rendering quality.

Problem

• real-to-sim reconstruction
• urban facade
• geometry stability
• embodied AI
• 3D reconstruction

Innovation

• Shell-Supervised Gaussian Splatting
• Urban Real-to-Sim Reconstruction
• Geometric Regularization
• Facade Structure Prior
• Mask-Gated Optimization