Urban Socio-Semantic Segmentation with Vision-Language Reasoning

2026-01-15

AI Summary

This work addresses the limitation of existing remote sensing image segmentation methods, which primarily focus on physically defined objects (e.g., buildings, water bodies) and struggle to recognize socially defined semantic entities such as schools or parks. To bridge this gap, the study introduces a novel urban socio-semantic segmentation task and presents SocioSeg, a new dataset featuring hierarchical annotations. Furthermore, it proposes SocioReasoner, a reinforcement learning–based, non-differentiable vision-language reasoning framework that emulates human-like multi-stage cross-modal inference. By effectively integrating satellite imagery, digital maps, and pixel-level semantics, SocioReasoner substantially outperforms existing models on SocioSeg and demonstrates exceptional zero-shot generalization capabilities, offering a new paradigm for urban social perception through remote sensing.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in https://github.com/AMAP-ML/SocioReasoner.

Problem

• socio-semantic segmentation
• satellite imagery
• social semantic entities
• urban segmentation
• vision-language reasoning

Innovation

• vision-language reasoning
• socio-semantic segmentation
• reinforcement learning
• cross-modal recognition
• zero-shot generalization