Constructing a 3D Town from a Single Image

2025-05-21

AI Summary

This paper addresses key challenges—geometric inconsistency, layout hallucination, and low mesh quality—in generating full-scene 3D urban environments from a single overhead image. We propose a training-free, 3D-supervision-free end-to-end framework. Methodologically, we introduce the novel “region decomposition + spatially aware 3D refinement” paradigm: the overhead image is partitioned into overlapping regions; a pre-trained 3D object generator initializes local structures, and mask-guided rectified flow performs cross-region geometric alignment and texture refinement to ensure global coherence and detail fidelity. Our core contribution lies in overcoming geometric distortion and resolution limitations inherent in single-image 3D generation, enabling high-fidelity scene reconstruction without training or 3D annotations. Experiments demonstrate that our method consistently outperforms state-of-the-art approaches—including Trellis, Hunyuan3D-2, and TripoSG—in geometric accuracy, spatial consistency, and texture quality.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce 3DTown, a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that 3DTown outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach.

Problem

• Generating 3D scenes from single top-down images
• Overcoming inconsistent geometry and layout hallucinations
• Ensuring high-quality meshes without 3D supervision

Innovation

• Training-free framework for 3D scene synthesis
• Region-based generation for better alignment
• Spatial-aware 3D inpainting for scene coherence