SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images

2025-12-09

AI Summary

Current CLIP-based, training-free open-vocabulary semantic segmentation (OVSS) methods for remote sensing imagery suffer from inaccurate localization of dense small objects and redundant inference pipelines. To address these issues, this paper proposes an end-to-end zero-shot OVSS framework tailored for remote sensing. Its core contributions are threefold: (1) the first integration of SAM 3 into remote sensing OVSS, jointly leveraging its semantic and instance heads for unified feature representation; (2) a novel mask fusion strategy to improve geographical object coverage completeness; and (3) an existence-score filtering mechanism to suppress spurious class predictions and enhance classification accuracy. Crucially, the method requires no training or fine-tuning. Extensive experiments across multiple remote sensing benchmarks demonstrate significant reductions in false detection rates, validating SAM 3’s effectiveness and generalization capability for high-precision, robust open-vocabulary segmentation in complex geospatial scenes.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Semi-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance

Abstract

Most existing methods for training-free Open-Vocabulary Semantic Segmentation (OVSS) are based on CLIP. While these approaches have made progress, they often face challenges in precise localization or require complex pipelines to combine separate modules, especially in remote sensing scenarios where numerous dense and small targets are present. Recently, Segment Anything Model 3 (SAM 3) was proposed, unifying segmentation and recognition in a promptable framework. In this paper, we present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training. First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder (instance head). This allows us to leverage the strengths of both heads for better land coverage. Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene, reducing false positives caused by the vast vocabulary sizes and patch-level processing in geospatial scenes. We evaluate our method on extensive remote sensing datasets. Experiments show that this simple adaptation achieves promising performance, demonstrating the potential of SAM 3 for remote sensing OVSS. Our code is released at https://github.com/earth-insights/SegEarth-OV-3.

Problem

• Apply SAM 3 to remote sensing open-vocabulary semantic segmentation without training
• Improve segmentation accuracy for dense, small targets in remote sensing images
• Reduce false positives from large vocabularies in geospatial scene analysis

Innovation

• Uses SAM 3 for training-free open-vocabulary segmentation
• Implements mask fusion from semantic and instance heads
• Filters categories with presence scores to reduce false positives