GeoSeg-OV: Bridging Geospatial Gaps with Structural Guidance for Open-Vocabulary Remote Sensing Segmentation

2026-08-10

AI Summary

This work addresses the challenges of weak visual-text alignment and limited cross-domain generalization in open-vocabulary remote sensing image segmentation, which arise from variations in geographic regions, spatial resolutions, and acquisition platforms. To overcome these issues, the authors propose a Structure-Guided Aggregation (SGA) mechanism and a Cost-aware Decoding (CAD) strategy. By decoupling structural features from a frozen Vision Foundation Model (VFM) and using them as spatial propagation biases, they construct a rotation-robust multi-orientation CLIP cost volume. This enables semantic-structural co-guided pixel-level segmentation through multi-scale structure-sensitive features, while avoiding direct involvement of the VFM in visual-text matching. Evaluated on the HRLC benchmark spanning seven datasets across six continents, the method achieves average mIoU improvements of 2.5 and 2.7 under two training settings, with zero-shot experiments demonstrating effective cross-regional and cross-taxonomy generalization without target-domain annotations.

Abstract

Open-vocabulary remote sensing segmentation has recently emerged as a promising paradigm that enables pixel-level recognition of arbitrary categories specified by natural language, including classes unseen during training. However, geospatial domain shifts caused by heterogeneous regions, spatial resolutions, and acquisition platforms weaken visual-text matching and limit cross-dataset generalization. Recent attempts have begun to incorporate auxiliary vision foundation models (VFMs), typically coupling their features with text embeddings as additional matching evidence. However, this strategy may introduce inconsistent matching signals while leaving the structure-sensitive representations of VFMs insufficiently exploited. We therefore propose GeoSeg-OV, which decouples auxiliary VFM features from visual-text matching and repurposes them as structural guidance for cost aggregation and decoding. GeoSeg-OV constructs an orientation-robust cost volume from multi-rotation CLIP features, while a frozen VFM extracts multi-scale structure-sensitive features in parallel. We propose Structure-Guided Aggregation (SGA), which integrates cost tokens and CLIP semantic guidance with VFM-derived pairwise structural biases for coherent spatial propagation, followed by text-conditioned class-wise reasoning. We further introduce Cost-Aware Decoding (CAD) to adaptively refine and fuse multi-scale semantic and structural guidance based on the current decoder context. On the global High-Resolution Land Cover (HRLC) benchmark spanning seven datasets across six continents, GeoSeg-OV outperforms the state-of-the-art by +2.5 and +2.7 average mIoU under two training settings. A large-scale zero-shot case study further demonstrates its generalization across geographic domains and category systems without target-domain annotations or retraining.

Problem

• open-vocabulary segmentation
• geospatial domain shift
• remote sensing
• cross-dataset generalization
• visual-text matching

Innovation

• open-vocabulary segmentation
• vision foundation model
• structural guidance
• cost aggregation
• zero-shot generalization