EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion

2025-07-22

AI Summary

Existing 3D generative methods struggle to scale to geospatial extents (e.g., thousands of square kilometers). This work introduces the first generative framework tailored for large-scale Earth surface modeling. Methodologically, it proposes a dual-sparse latent diffusion architecture that decouples geometric structure modeling (via volumetric voxels) from texture synthesis (via 2D Gaussians); constructs Aerial-Earth3D—the first large-scale aerial photogrammetric 3D dataset with semantic annotations; and incorporates a conditional-aware flow matching model enabling both semantic-guided generation (from semantic maps or aerial imagery) and unconditional synthesis. The framework achieves efficient, high-fidelity 3D Earth generation while preserving geographic plausibility, significantly outperforming prior approaches in fidelity, scalability, and semantic controllability.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the remarkable developments achieved by recent 3D generation works, scaling these methods to geographic extents, such as modeling thousands of square kilometers of Earth's surface, remains an open challenge. We address this through a dual innovation in data infrastructure and model architecture. First, we introduce Aerial-Earth3D, the largest 3D aerial dataset to date, consisting of 50k curated scenes (each measuring 600m x 600m) captured across the U.S. mainland, comprising 45M multi-view Google Earth frames. Each scene provides pose-annotated multi-view images, depth maps, normals, semantic segmentation, and camera poses, with explicit quality control to ensure terrain diversity. Building on this foundation, we propose EarthCrafter, a tailored framework for large-scale 3D Earth generation via sparse-decoupled latent diffusion. Our architecture separates structural and textural generation: 1) Dual sparse 3D-VAEs compress high-resolution geometric voxels and textural 2D Gaussian Splats (2DGS) into compact latent spaces, largely alleviating the costly computation suffering from vast geographic scales while preserving critical information. 2) We propose condition-aware flow matching models trained on mixed inputs (semantics, images, or neither) to flexibly model latent geometry and texture features independently. Extensive experiments demonstrate that EarthCrafter performs substantially better in extremely large-scale generation. The framework further supports versatile applications, from semantic-guided urban layout generation to unconditional terrain synthesis, while maintaining geographic plausibility through our rich data priors from Aerial-Earth3D.

Problem

• Scaling 3D generation to large geographic extents
• Creating a dataset for diverse 3D Earth modeling
• Separating structural and textural generation for efficiency

Innovation

• Largest 3D aerial dataset Aerial-Earth3D
• Dual sparse 3D-VAEs for efficient compression
• Condition-aware flow matching for flexible modeling