Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation

2026-06-20

AI Summary

This work addresses key challenges in open-vocabulary semantic segmentation of remote sensing imagery—namely, insufficient coverage from single prompts, redundancy and noise propagation in online multi-prompt encoding—by proposing an efficient framework for calibrating the Segment Anything Model (SAM). The approach constructs an offline high-quality prompt pool through a class-aware matcher with expansion constraints, caches text embeddings to avoid redundant computation, and introduces an existence-guided residual fusion mechanism coupled with a peak-preserving class aggregation strategy. These components collectively enhance segmentation performance, particularly for small and sparsely distributed objects. Evaluated across eight remote sensing benchmarks, the method achieves an average mIoU of 56.1%, outperforming the current best training-free approach by 3.9 percentage points.

Abstract

Open-vocabulary semantic segmentation (OVSS) in remote sensing images aims to segment categories beyond a fixed label space. Recent SAM 3-based methods provide a promising training-free foundation, yet three key issues remain: (1) a single class-name prompt lacks sufficient semantic coverage for complex remote sensing categories; (2) expanding each category into multiple prompts introduces redundant online text encoding; and (3) directly aggregating multiple prompt responses propagates noisy activations into the final prediction. To address these issues, we propose ProC-SAM3, which calibrates SAM 3's prompt interface for remote sensing OVSS from three complementary aspects. First, we construct an offline prompt pool where a Category Matcher groups MLLM-generated candidates into per-category sets, and Expansion Constraints further refine each set using category-specific prior knowledge. Second, the resulting text embeddings are cached and reused across all test images, eliminating repeated text encoding. Third, we introduce Presence-Guided Residual Fusion to gate unreliable decoder outputs by prompt presence and confidence, followed by peak-preserving class aggregation that retains fine-grained activations for small and sparse objects. Experiments on eight benchmarks show that ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points. Code will be available at https://github.com/YanghuiSong/ProC-SAM3.

Problem

• open-vocabulary semantic segmentation
• remote sensing
• prompt calibration
• semantic coverage
• noisy activation

Innovation

• Prompt Calibration
• Open-Vocabulary Segmentation
• Remote Sensing
• SAM3
• Text Embedding Caching