ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures

2026-06-26

AI Summary

Constructing structured indoor 3D scenes suitable for simulation is challenged by missing object-level structure, difficulty in integrating cross-view relationships, and physical inconsistency. This work proposes ReScene, a framework that leverages multi-view geometry as a unified prior. It introduces the HierView module to select views that are semantically consistent and provide complete 3D coverage, and employs Relation-Aware Assembly to integrate relationships predicted by vision-language models with room-shell geometric priors, enabling, for the first time, the construction of physically plausible scene graphs. Departing from conventional max-mask heuristics, ReScene adopts a semantics-geometry co-driven view selection mechanism, significantly outperforming existing methods on ScanNet—reducing Chamfer Distance by 17% and LPIPS by 26%, while achieving a 10× faster inference speed—and facilitating high-quality embodied visual question answering data generation.

Abstract

Constructing simulation-ready 3D scenes from multi-view captures is a key bottleneck for Embodied Artificial Intelligence, as downstream tasks require object-level structure, explicit inter-object relations, and physical plausibility. Existing approaches either rely on specialized capture hardware, suffer from single-view bias in object reconstruction, or yield layouts that are geometrically reasonable but physically inconsistent. We identify that the problem is not single-object reconstruction but cross-view relation fusion and physically plausible scene assembly. To address this challenge, we present ReScene, a framework that threads multi-view geometry throughout the pipeline as a unifying prior. Our method consists of two main components: HierView prioritizes reconstruction views based on semantic consistency and 3D coverage completeness, replacing the largest-mask heuristic that conflates image occupancy with object coverage; and Relation-Aware Assembly fuses multi-frame relation predictions from a vision-language model with geometric and room-shell priors into a confidence-weighted scene graph, enabling physically consistent scene assembly. ReScene sets a new state of the art across geometry, rendering, and perceptual quality on a set of ScanNet scenes, achieving a 17% reduction in Chamfer Distance and 26% in LPIPS over the strongest prior baseline, while running up to 10x faster than prior multi-view methods. Based on the reconstructed scenes, we also generate an embodied visual question answering dataset, on which fine-tuned Qwen-VL approaches the performance of strong closed-source models on several spatial reasoning tasks.

Problem

• scene reconstruction
• multi-view capture
• physical plausibility
• object relations
• embodied AI

Innovation

• multi-view reconstruction
• scene graph
• physical plausibility
• relation-aware assembly
• structured 3D scene