WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence

2026-07-07

AI Summary

Existing AI systems struggle to construct effective spatial representations at real-world urban scales, primarily due to the lack of large-scale, multimodal, and in-the-wild urban scene data. To address this limitation, this work leverages autonomous driving fleets to collect multisensor trajectory data across complex urban environments, establishing the first city-scale testbed capable of supporting rendering, simulation, and spatial intelligence research. We release a large-scale dataset comprising 18 trajectories with an average length of 83.7 kilometers, introduce city-customized reconstruction baselines and a closed-loop simulation environment, and systematically analyze the key challenges in building simulation-ready digital twins. This effort significantly advances AI’s capabilities in urban-scale perception, memory, and spatial reasoning.

Abstract

Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments. Our dataset includes 18 trajectories, each averaging 83.7 kilometers in length, and preserves the core challenges of in-the-wild perception, e.g., dynamic objects, lighting variations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environments into a closed-loop simulator. Beyond the dataset and baseline, we systematically analyze the key challenges on the path to simulation-ready urban digital twins: scalability, extrapolation, and uncertainty. Ultimately, WildCity aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition. Project page: https://han-xiangyu.github.io/Wild-City/

Problem

• city-scale spatial intelligence
• urban digital twins
• spatial representation
• real-world multimodal data
• embodied AI

Innovation

• city-scale dataset
• spatial intelligence
• urban digital twin
• multimodal perception
• closed-loop simulation