SAM 3: Segment Anything with Concepts

2025-11-20

AI Summary

This paper introduces Promptable Concept Segmentation (PCS), a novel task that unifies object detection, segmentation, and tracking in images and videos via concept prompts—such as noun phrases or exemplar images. Methodologically: (1) we construct a large-scale dataset with 4 million concept labels, decoupling recognition from localization and introducing an existence prediction head; (2) we design a shared single-backbone image detector and a memory-based video tracker, integrating concept prompting mechanisms with a high-quality data engine. Our contributions are threefold: first, we formally define the PCS task and establish SA-Co, a new benchmark for it; second, our method achieves double the accuracy of prior state-of-the-art methods on both image and video PCS benchmarks; third, it surpasses SAM on multiple segmentation benchmarks. All models and benchmarks are publicly released.

Technology Category

• Computer Vision: Segmentation
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.

Problem

• Detects segments and tracks objects using concept prompts
• Creates scalable dataset with 4M unique concept labels
• Improves segmentation accuracy while decoupling recognition localization

Innovation

• Uses concept prompts for object detection and segmentation
• Implements scalable data engine with 4M concept labels
• Decouples recognition and localization with presence head