Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation

2026-02-06

AI Summary

This work addresses the degradation of vision-language prompt alignment in SAM3 under open-vocabulary segmentation due to data and concept drift. To tackle this issue, the authors propose ConceptBank, a dynamic calibration framework that operates without requiring parameter updates. ConceptBank constructs a target-domain-specific concept bank based on statistical characteristics, leveraging class-level visual prototypes as anchors, mining representative support samples, and fusing multiple concepts to jointly correct distribution shifts and restore prompt alignment. Notably, ConceptBank adapts seamlessly to new domains without fine-tuning and significantly enhances the robustness and efficiency of SAM3 in challenging scenarios such as natural scenes and remote sensing imagery, thereby establishing a new benchmark for open-vocabulary segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The recent introduction of \texttt{SAM3} has revolutionized Open-Vocabulary Segmentation (OVS) through \textit{promptable concept segmentation}, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (\textit{data drift}) or conditional label distributions evolve (\textit{concept drift}) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present \textsc{ConceptBank}, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (\textit{i}) anchors target-domain evidence via class-wise visual prototypes, (\textit{ii}) mines representative supports to suppress outliers under data drift, and (\textit{iii}) fuses candidate concepts to rectify concept drift. We demonstrate that \textsc{ConceptBank} effectively adapts \texttt{SAM3} to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.

Problem

• Open-Vocabulary Segmentation
• data drift
• concept drift
• promptable concept segmentation
• distribution shift

Innovation

• ConceptBank
• Open-Vocabulary Segmentation
• Data Drift
• Concept Drift
• Promptable Segmentation