Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline

2026-04-16

AI Summary

This work addresses the lack of large-scale, diverse evaluation benchmarks for open-vocabulary remote sensing image segmentation tailored to real-world geospatial applications. To this end, we introduce OVRSISBenchV2, comprising 95K image-mask pairs across 128 semantic categories, and establish the first comprehensive evaluation framework covering downstream tasks such as building extraction, road extraction, and flood detection. We also propose Pi-Seg, a novel baseline model that incorporates a semantics-guided positive-incentive noise mechanism to enhance visual-textual feature transfer. Extensive experiments demonstrate that Pi-Seg achieves robust performance on OVRSISBenchV2 and multiple cross-dataset downstream tasks, significantly outperforming existing methods in complex real-world scenarios.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Security and Privacy: Data transparency and provenance

Abstract

Open-vocabulary remote sensing image segmentation (OVRSIS) remains underexplored due to fragmented datasets, limited training diversity, and the lack of evaluation benchmarks that reflect realistic geospatial application demands. Our previous \textit{OVRSISBenchV1} established an initial cross-dataset evaluation protocol, but its limited scope is insufficient for assessing realistic open-world generalization. To address this issue, we propose \textit{OVRSISBenchV2}, a large-scale and application-oriented benchmark for OVRSIS. We first construct \textbf{OVRSIS95K}, a balanced dataset of about 95K image--mask pairs covering 35 common semantic categories across diverse remote sensing scenes. Built upon OVRSIS95K and 10 downstream datasets, OVRSISBenchV2 contains 170K images and 128 categories, substantially expanding scene diversity, semantic coverage, and evaluation difficulty. Beyond standard open-vocabulary segmentation, it further includes downstream protocols for building extraction, road extraction, and flood detection, thereby better reflecting realistic geospatial application demands and complex deployment scenarios. We also propose \textbf{Pi-Seg}, a baseline for OVRSIS. Pi-Seg improves transferability through a \textbf{positive-incentive noise} mechanism, where learnable and semantically guided perturbations broaden the visual-text feature space during training. Extensive experiments on OVRSISBenchV1, OVRSISBenchV2, and downstream tasks show that Pi-Seg delivers strong and consistent results, particularly on the more challenging OVRSISBenchV2 benchmark. Our results highlight both the importance of realistic benchmark design and the effectiveness of perturbation-based transfer for OVRSIS. The code and datasets are available at \href{https://github.com/LiBingyu01/RSKT-Seg/tree/Pi-Seg}{LiBingyu01/RSKT-Seg/tree/Pi-Seg}.

Problem

• open-vocabulary segmentation
• remote sensing
• benchmark
• geospatial applications
• realistic evaluation

Innovation

• open-vocabulary segmentation
• remote sensing
• benchmark
• positive-incentive noise
• transferability