Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing

2026-07-10

AI Summary

This work addresses the limited generalization of SAM 3 to complex overhead geometric structures in remote sensing imagery under zero- and one-shot settings. It proposes a training-agnostic, universal proxy evaluation protocol for zero-shot assessment, empirically evaluating performance across scene classification, object detection, and instance segmentation through multitask experiments. The binary existence head is reformulated as a zero-shot classifier to systematically analyze the alignment mechanism between textual and visual prompts within the multimodal decoder. The study reveals, for the first time, that textual prompts introduce ground-level semantic biases that degrade coordinate regression and identifies cross-modal interference in SAM 3 when applied to remote sensing. Experiments show that SAM 3 achieves high harmonic mean scores in segmentation without overfitting, yet its performance is constrained by sub-pixel resolution limits and aerial semantic blind spots, highlighting the need for parameter-efficient fine-tuning of its multimodal decoder.

Abstract

The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3's performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model's multimodal decoder. Our findings reveal severe cross-modal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder.

Problem

• foundation models
• remote sensing
• zero-shot learning
• multimodal alignment
• instance segmentation

Innovation

• zero-shot segmentation
• multimodal prompting
• remote sensing
• foundation model adaptation
• training-free evaluation