CityRAG: Stepping Into a City via Spatially-Grounded Video Generation

2026-04-21

AI Summary

This work proposes a method for generating spatially consistent, navigable, and geographically grounded 3D video environments to support autonomous driving and robotics simulation. By fusing georegistered, multi-source, temporally misaligned data, the model achieves semantic disentanglement between static scenes and dynamic attributes during training and incorporates spatiotemporal priors to enable, for the first time, coherent long-horizon video synthesis under closed-loop trajectories and diverse weather conditions. The approach successfully generates physically plausible video sequences exceeding one minute in duration (over one thousand frames), significantly outperforming existing methods in complex trajectory navigation, illumination and weather consistency, and fidelity to real-world geographic reconstruction.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We address the problem of generating a 3D-consistent, navigable environment that is spatially grounded: a simulation of a real location. Existing video generative models can produce a plausible sequence that is consistent with a text (T2V) or image (I2V) prompt. However, the capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation. To this end, we present CityRAG, a video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene, while maintaining learned priors for complex motion and appearance changes. CityRAG relies on temporally unaligned training data, which teaches the model to semantically disentangle the underlying scene from its transient attributes. Our experiments demonstrate that CityRAG can generate coherent minutes-long, physically grounded video sequences, maintain weather and lighting conditions over thousands of frames, achieve loop closure, and navigate complex trajectories to reconstruct real-world geography.

Problem

• spatially-grounded video generation
• 3D-consistent environment
• real-world reconstruction
• navigable simulation
• geo-registered data

Innovation

• spatially-grounded video generation
• geo-registered data
• semantic disentanglement
• 3D-consistent navigation
• real-world simulation