PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models

2025-05-20

AI Summary

Existing vision-language models (VLMs) struggle to interpret urban planning maps—such as land-use layouts and functional zoning—due to their intricate spatial configurations, regulatory constraints, and multi-scale semantic structures. Method: We introduce PlanAnno-V, the first domain-specific VLM for planning map understanding, built upon a novel data synthesis framework. It incorporates keypoint-based reasoning to mitigate hallucination and adopts an efficient frozen-vision-encoder fine-tuning paradigm. Our approach integrates spatial-semantic alignment, multi-scale map comprehension, and structured verification reasoning. Contribution/Results: Evaluated on our newly constructed benchmark PlanBench-V, the 7B-parameter model achieves performance on par with a 72B general-purpose VLM, with substantially improved factual accuracy. The model has been successfully deployed in real-world urban planning analysis and pedagogical applications.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In the field of urban planning, existing Vision-Language Models (VLMs) frequently fail to effectively analyze and evaluate planning maps, despite the critical importance of these visual elements for urban planners and related educational contexts. Planning maps, which visualize land use, infrastructure layouts, and functional zoning, require specialized understanding of spatial configurations, regulatory requirements, and multi-scale analysis. To address this challenge, we introduce PlanGPT-VL, the first domain-specific Vision-Language Model tailored specifically for urban planning maps. PlanGPT-VL employs three innovative approaches: (1) PlanAnno-V framework for high-quality VQA data synthesis, (2) Critical Point Thinking to reduce hallucinations through structured verification, and (3) comprehensive training methodology combining Supervised Fine-Tuning with frozen vision encoder parameters. Through systematic evaluation on our proposed PlanBench-V benchmark, we demonstrate that PlanGPT-VL significantly outperforms general-purpose state-of-the-art VLMs in specialized planning map interpretation tasks, offering urban planning professionals a reliable tool for map analysis, assessment, and educational applications while maintaining high factual accuracy. Our lightweight 7B parameter model achieves comparable performance to models exceeding 72B parameters, demonstrating efficient domain specialization without sacrificing performance.

Problem

• Existing VLMs fail to analyze urban planning maps effectively
• Specialized understanding of spatial configurations is required
• PlanGPT-VL outperforms general VLMs in planning map tasks

Innovation

• PlanAnno-V framework for VQA data synthesis
• Critical Point Thinking reduces hallucinations
• Supervised Fine-Tuning with frozen vision encoder