DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes

2024-10-23

AI Summary

Existing methods predominantly generate static, single-frame urban scenes, failing to capture the dynamic spatiotemporal characteristics of real-world driving environments. To address this, we propose the first generative framework for large-scale semantic 4D (spatiotemporal) occupancy scene synthesis, breaking the static modeling bottleneck. Our core contributions are: (1) HexPlane—a compact 4D representation—coupled with a Projection Module for efficient feature compression; (2) an Expansion&Squeeze parallel reconstruction strategy enabling high-fidelity 4D occupancy decoding; and (3) Padded Rollout, a novel operation that adapts HexPlane to DiT-based diffusion models and supports multi-condition controllable generation (e.g., trajectories, instructions, layouts). Evaluated on CarlaSC and Waymo, our method surpasses SOTA: achieving up to +12.56% mIoU, 2.06× faster training, and 70.84% reduced GPU memory consumption. It further enables diverse applications including 4D scene completion and instruction-driven generation.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion&Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.

Problem

• Generates large-scale dynamic 4D urban scenes
• Improves 4D feature compression and reconstruction accuracy
• Supports versatile 4D generation applications with conditions

Innovation

• VAE model for compact 4D representation learning
• Projection Module compresses 4D features effectively
• DiT-based diffusion model for HexPlane generation