CityVerse: A Unified Data Platform for Multi-Task Urban Computing with Large Language Models

2025-11-13

AI Summary

To address the lack of unified benchmarking platforms and standardized task taxonomies for evaluating large language models (LLMs) in urban computing, this paper introduces UrbanLLM—the first open-source evaluation platform tailored for urban intelligence. Methodologically, it proposes a four-tier cognitive task taxonomy (perception → spatial understanding → reasoning & prediction → decision-making & interaction) to enable cross-modal, cross-task capability quantification; designs a coordinate-driven unified data interface integrating ten heterogeneous urban data categories (>38 million records); and incorporates dynamic simulation, interactive visualization, and hierarchical task APIs. Experimental evaluation systematically benchmarks state-of-the-art LLMs on representative urban tasks, supporting reproducible, extensible, multi-level capability assessment. UrbanLLM establishes a standardized infrastructure for urban LLM research, facilitating rigorous, comparable, and scalable evaluation across diverse urban intelligence scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) show remarkable potential for urban computing, from spatial reasoning to predictive analytics. However, evaluating LLMs across diverse urban tasks faces two critical challenges: lack of unified platforms for consistent multi-source data access and fragmented task definitions that hinder fair comparison. To address these challenges, we present CityVerse, the first unified platform integrating multi-source urban data, capability-based task taxonomy, and dynamic simulation for systematic LLM evaluation in urban contexts. CityVerse provides: 1) coordinate-based Data APIs unifying ten categories of urban data-including spatial features, temporal dynamics, demographics, and multi-modal imagery-with over 38 million curated records; 2) Task APIs organizing 43 urban computing tasks into a four-level cognitive hierarchy: Perception, Spatial Understanding, Reasoning and Prediction, and Decision and Interaction, enabling standardized evaluation across capability levels; 3) an interactive visualization frontend supporting real-time data retrieval, multi-layer display, and simulation replay for intuitive exploration and validation. We validate the platform's effectiveness through evaluations on mainstream LLMs across representative tasks, demonstrating its capability to support reproducible and systematic assessment. CityVerse provides a reusable foundation for advancing LLMs and multi-task approaches in the urban computing domain.

Problem

• Unified platform for multi-source urban data access
• Standardized task taxonomy enabling fair LLM comparisons
• Dynamic simulation supporting systematic urban computing evaluation

Innovation

• Unified platform integrating multi-source urban data
• Task APIs organizing urban tasks into cognitive hierarchy
• Interactive visualization frontend supporting real-time simulation