SAMST: A Transformer framework based on SAM pseudo label filtering for remote sensing semi-supervised semantic segmentation

2025-07-16

AI Summary

Remote sensing semantic segmentation faces two critical challenges: scarcity of annotated data and poor generalizability of public datasets—stemming from inter-dataset variations in spatial resolution and inconsistent class definitions. To address these, we propose SAMST, a semi-supervised framework that synergistically integrates the zero-shot boundary-aware capability of the Segment Anything Model (SAM) with the efficient trainability of lightweight segmentation networks. Methodologically, SAMST introduces three key components: threshold-based filtering, learnable prompt generation, and label stitching, enabling iterative refinement of pseudo-labels. Technically, it unifies a Transformer-based backbone, SAM-guided pseudo-label self-training, prompt learning, and connected-component optimization. Extensive evaluation on the Potsdam dataset demonstrates that SAMST significantly enhances pseudo-label quality and segmentation accuracy, effectively alleviating the annotation bottleneck in remote sensing imagery.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Public remote sensing datasets often face limitations in universality due to resolution variability and inconsistent land cover category definitions. To harness the vast pool of unlabeled remote sensing data, we propose SAMST, a semi-supervised semantic segmentation method. SAMST leverages the strengths of the Segment Anything Model (SAM) in zero-shot generalization and boundary detection. SAMST iteratively refines pseudo-labels through two main components: supervised model self-training using both labeled and pseudo-labeled data, and a SAM-based Pseudo-label Refiner. The Pseudo-label Refiner comprises three modules: a Threshold Filter Module for preprocessing, a Prompt Generation Module for extracting connected regions and generating prompts for SAM, and a Label Refinement Module for final label stitching. By integrating the generalization power of large models with the training efficiency of small models, SAMST improves pseudo-label accuracy, thereby enhancing overall model performance. Experiments on the Potsdam dataset validate the effectiveness and feasibility of SAMST, demonstrating its potential to address the challenges posed by limited labeled data in remote sensing semantic segmentation.

Problem

• Addresses limited universality in public remote sensing datasets
• Leverages unlabeled data for semi-supervised semantic segmentation
• Improves pseudo-label accuracy using SAM and iterative refinement

Innovation

• SAMST uses SAM for pseudo-label filtering
• Integrates self-training with SAM refinement
• Enhances pseudo-label accuracy efficiently